How difficult are exams? A framework for assessing the complexity of introductory programming exams

Student performance on examinations is influenced by the level of difficulty of the questions. It seems reasonable to propose therefore that assessment of the difficulty of exam questions could be used to gauge the level of skills and knowledge expected at the end of a course. This paper reports the results of a study investigating the difficulty of exam questions using a subjective assessment of difficulty and a purpose-built exam question complexity classification scheme. The scheme, devised for exams in introductory programming courses, assesses the complexity of each question using six measures: external domain references, explicitness, linguistic complexity, conceptual complexity, length of code involved in the question and/or answer, and intellectual complexity (Bloom level). We apply the scheme to 20 introductory programming exam papers from five countries, and find substantial variation across the exams for all measures. Most exams include a mix of questions of low, medium, and high difficulty, although seven of the 20 have no questions of high difficulty. All of the complexity measures correlate with assessment of difficulty, indicating that the difficulty of an exam question relates to each of these more specific measures. We discuss the implications of these findings for the development of measures to assess learning standards in programming courses.

A conceptual model for reflecting on expected learning vs. demonstrated student performance

Educators are faced with many challenging questions in designing an effective curriculum. What prerequisite knowledge do students have before commencing a new subject? At what level of mastery? What is the spread of capabilities between bare-passing students vs. the top performing group? How does the intended learning specification compare to student performance at the end of a subject? In this paper we present a conceptual model that helps in answering some of these questions. It has the following main capabilities: capturing the learning specification in terms of syllabus topics and outcomes; capturing mastery levels to model progression; capturing the minimal vs. aspirational learning design; capturing confidence and reliability metrics for each of these mappings; and finally, comparing and reflecting on the learning specification against actual student performance. We present a web-based implementation of the model, and validate it by mapping the final exams from four programming subjects against the ACM/IEEE CS2013 topics and outcomes, using Bloom's Taxonomy as the mastery scale. We then import the itemised exam grades from 632 students across the four subjects and compare the demonstrated student performance against the expected learning for each of these. Key contributions of this work are the validated conceptual model for capturing and comparing expected learning vs. demonstrated performance, and a web-based implementation of this model, which is made freely available online as a community resource.

On the reliability of classifying programming tasks using a Neo-Piagetian theory of cognitive development

Recent research has proposed Neo-Piagetian theory as a useful way of describing the cognitive development of novice programmers. Neo-Piagetian theory may also be a useful way to classify materials used in learning and assessment. If Neo-Piagetian coding of learning resources is to be useful then it is important that practitioners can learn it and apply it reliably. We describe the design of an interactive web-based tutorial for Neo-Piagetian categorization of assessment tasks. We also report an evaluation of the tutorial's effectiveness, in which twenty computer science educators participated. The average classification accuracy of the participants on each of the three Neo-Piagetian stages were 85%, 71% and 78%. Participants also rated their agreement with the expert classifications, and indicated high agreement (91%, 83% and 91% across the three Neo-Piagetian stages). Self-rated confidence in applying Neo-Piagetian theory to classifying programming questions before and after the tutorial were 29% and 75% respectively. Our key contribution is the demonstration of the feasibility of the Neo-Piagetian approach to classifying assessment materials, by demonstrating that it is learnable and can be applied reliably by a group of educators. Our tutorial is freely available as a community resource.

Intelligent tutoring system for learning PHP

This thesis investigates the possibility of using an adaptive tutoring system for beginning programming students. The work involved, designing, developing and evaluating such a system and showing that it was effective in increasing the students’ test scores. In doing so, Artificial Intelligence techniques were used to analyse PHP programs written by students and to provide feedback based on any specific errors made by them. Methods were also included to provide students with the next best exercise to suit their particular level of knowledge.

Incorporating anchored learning in a C# Intelligent Tutoring System

Learning programming is known to be difficult. One possible reason why students fail programming is related to the fact that traditional learning in the classroom places more emphasis on lecturing the material instead of applying the material to a real application. For some students, this teaching model may not catch their interest. As a result they may not give their best effort to understand the material given. Seeing how the knowledge can be applied to real life problems can increase student interest in learning. As a consequence, this will increase their effort to learn. Anchored learning that applies knowledge to solve real life problems may be the key to improving student performance. In anchored learning, it is necessary to provide resources that can be accessed by the student as they learn. These resources can be provided by creating an Intelligent Tutoring System (ITS) that can support the student when they need help or experience a problem. Unfortunately, there is no ITS developed for the programming domain that has incorporated anchored learning in its teaching system. Having an ITS that supports anchored learning will not only be able to help the student learn programming effectively but will also make the learning process more enjoyable. This research tries to help students learn C# programming using an anchored learning ITS named CSTutor. Role playing is used in CSTutor to present a real world situation where they develop their skills. A knowledge base using First Order Logic is used to represent the student's code and to give feedback and assistance accordingly.

The QUT innovation space : a trans-disciplinary learning environment for entrepreneurship education : final report

In the current climate of global economic volatility, there are increasing calls for training in enterprising skills and entrepreneurship to underpin the systemic innovation required for even medium-term business sustainability. The skills long-recognised as the essential for entrepreneurship now appear on the list of employability skills demanded by industry. The QUT Innovation Space (QIS) was an experiment aimed at delivering entrepreneurship education (EE), as an extra-curricular platform across the university, to the undergraduate students of an Australian higher education institute. It was an ambitious project that built on overseas models of EE studied during an Australian Learning and Teaching Council (ALTC) Teaching Fellowship (Collet, 2011) and implemented those approaches across an institute. Such EE approaches have not been attempted in an Australian university. The project tested resonance not only with the student population, from the perspective of what worked and what didn’t work, but also with every level of university operations. Such information is needed to inform the development of EE in the Australian university landscape. The QIS comprised a physical co-working space, virtual sites (web, Twitter and Facebook) and a network of entrepreneurial mentors, colleagues, and students. All facets of the QIS enabled connection between like-minded individuals that underpins the momentum needed for a project of this nature. The QIS became an innovation community within QUT. This report serves two purposes. First, as an account of the QIS project and its evolution, the report serves to identify the student demand for skills and training as well as barriers and facilitators of the activities that promote EE in an Australian university context. Second, the report serves as a how-to manual, in the tradition of many tomes on EE, outlining the QIS activities that worked as well as those that failed. The activities represent one measure of QIS outcomes and are described herein to facilitate implementation in other institutes. The QIS initially aimed to adopt an incubation model for training in EE. The ‘learning by doing’ model for new venture creation is a highly successful and high profile training approach commonly found in overseas contexts. However, the greatest demand of the QUT student population was not for incubation and progression of a developed entrepreneurial intent, but rather for training that instilled enterprising skills in the individual. These two scenarios require different training approaches (Fayolle and Gailly, 2008). The activities of the QIS evolved to meet that student demand. In addressing enterprising skills, the QIS developed the antecedents of entrepreneurialism (i.e., entrepreneurial attitudes, motivation and behaviours) including high-level skills around risk-taking, effective communication, opportunity recognition and action-orientation. In focusing on the would-be entrepreneur and not on the (initial) idea per se, the QIS also fostered entrepreneurial outcomes that would never have gained entry to the rigid stage-gated incubation model proposed for the original QIS framework. Important lessons learned from the project for development of an innovation community include the need to: 1. Evaluate the context of the type of EE program to be delivered and the student demand for the skills training (as noted above). 2. Create a community that builds on three dimensions: a physical space, a virtual environment and a network of mentors and partners. 3. Supplement the community with external partnerships that aid in delivery of skills training materials. 4. Ensure discovery of the community through the use of external IT services to deliver advertising and networking outlets. 5. Manage unrealistic student expectations of billion dollar products. 6. Continuously renew and rebuild simple activities to maintain student engagement. 7. Accommodate the non-university end-user group within the community. 8. Recognise and address the skills bottlenecks that serve as barriers to concept progression; in this case, externally provided IT and programming skills. 9. Use available on-line and published resources rather than engage in constructing project-specific resources that quickly become obsolete. 10. Avoid perceptions of faculty ownership and operate in an increasingly competitive environment. 11. Recognise that the continuum between creativity/innovation and entrepreneurship is complex, non-linear and requires different training regimes during the different phases of the pipeline. One small entity, such as the QIS, cannot address them all. The QIS successfully designed, implemented and delivered activities that included events, workshops, seminars and services to QUT students in the extra-curricular space. That the QIS project can be considered successful derives directly from the outcomes. First, the QIS project changed the lives of emerging QUT student entrepreneurs. Also, the QIS activities developed enterprising skills in students who did not necessarily have a business proposition, at the time. Second, successful outcomes of the QIS project are evidenced as the embedding of most, perhaps all, of the QIS activities in a new Chancellery-sponsored initiative: the Leadership Development and Innovation Program hosted by QUT Student Support Services. During the course of the QIS project, the Brisbane-based innovation ecosystem underwent substantial change. From a dearth of opportunities for the entrepreneurially inclined, there is now a plethora of entities that cater for a diversity of innovation-related activities. While the QIS evolved with the landscape, the demand endpoint of the QIS activities still highlights a gap in the local and national innovation ecosystems. The freedom to experiment and to fail is not catered for by the many new entities seeking to build viable businesses on the back of the innovation push. The onus of teaching the enterprising skills, which are the employability skills now demanded by industry, remains the domain of the higher education sector.

Incorporating anchored learning in a C# intelligent tutoring system

This research showed that one solution that can be used to help the students learn how to program is by providing a system that can behave like a tutor to teach the students individually. An intelligent tutoring system named CSTutor was built in this research to assist the students. CSTutor asks the student to write programs in a role playing environment, presenting the most appropriate tasks to the students, and provides help to the students' problems.

Managing and learning with multiple models: Objectives and optimization algorithms

The quality of environmental decisions should be gauged according to managers' objectives. Management objectives generally seek to maximize quantifiable measures of system benefit, for instance population growth rate. Reaching these goals often requires a certain degree of learning about the system. Learning can occur by using management action in combination with a monitoring system. Furthermore, actions can be chosen strategically to obtain specific kinds of information. Formal decision making tools can choose actions to favor such learning in two ways: implicitly via the optimization algorithm that is used when there is a management objective (for instance, when using adaptive management), or explicitly by quantifying knowledge and using it as the fundamental project objective, an approach new to conservation.This paper outlines three conservation project objectives - a pure management objective, a pure learning objective, and an objective that is a weighted mixture of these two. We use eight optimization algorithms to choose actions that meet project objectives and illustrate them in a simulated conservation project. The algorithms provide a taxonomy of decision making tools in conservation management when there is uncertainty surrounding competing models of system function. The algorithms build upon each other such that their differences are highlighted and practitioners may see where their decision making tools can be improved. © 2010 Elsevier Ltd.

The use of hand-held mobile devices as aids to learning within flexible learning spaces in tertiary Information Technology classes

Mobile devices are very popular among tertiary student populations. This study looks at student use of hand-held mobile devices within the context of a first year programming unit. This research sought for ways in which an educational app on these devices could be successfully integrated into such a class's learning.

Programming: Reading, writing and reversing

In this paper, we look at the concept of reversibility, that is, negating opposites, counterbalances, and actions that can be reversed. Piaget identified reversibility as an indicator of the ability to reason at a concrete operational level. We investigate to what degree novice programmers manifest the ability to work with this concept of reversibility by providing them with a small piece of code and then asking them to write code that undoes the effect of that code. On testing entire cohorts of students in their first year of learning to program, we found an overwhelming majority of them could not cope with such a concept. We then conducted think aloud studies of novices where we observed them working on this task and analyzed their contrasting abilities to deal with it. The results of this study demonstrate the need for better understanding our students' reasoning abilities, and a teaching model aimed at that level of reality.

Mathematics, programming, and STEM

Learning mathematics is a complex and dynamic process. In this paper, the authors adopt a semiotic framework (Yeh & Nason, 2004) and highlight programming as one of the main aspects of the semiosis or meaning-making for the learning of mathematics. During a 10-week teaching experiment, mathematical meaning-making was enriched when primary students wrote Logo programs to create 3D virtual worlds. The analysis of results found deep learning in mathematics, as well as in technology and engineering areas. This prompted a rethinking about the nature of learning mathematics and a need to employ and examine a more holistic learning approach for the learning in science, technology, engineering, and mathematics (STEM) areas.

A model eliciting framework for integrating mathematics and robotics learning

Robotics is taught in many Australian ICT classrooms, in both primary and secondary schools. Robotics activities, including those developed using the LEGO Mindstorms NXT technology, are mathematics-rich and provide a fertile round for learners to develop and extend their mathematical thinking. However, this context for learning mathematics is often under-exploited. In this paper a variant of the model construction sequence (Lesh, Cramer, Doerr, Post, & Zawojewski, 2003) is proposed, with the purpose of explicitly integrating robotics and mathematics teaching and learning. Lesh et al.’s model construction sequence and the model eliciting activities it embeds were initially researched in primary mathematics classrooms and more recently in university engineering courses. The model construction sequence involves learners working collaboratively upon product-focussed tasks, through which they develop and expose their conceptual understanding. The integrating model proposed in this paper has been used to design and analyse a sequence of activities in an Australian Year 4 classroom. In that sequence more traditional classroom learning was complemented by the programming of LEGO-based robots to ‘act out’ the addition and subtraction of simple fractions (tenths) on a number-line. The framework was found to be useful for planning the sequence of learning and, more importantly, provided the participating teacher with the ability to critically reflect upon robotics technology as a tool to scaffold the learning of mathematics.