How difficult are exams? A framework for assessing the complexity of introductory programming exams

Student performance on examinations is influenced by the level of difficulty of the questions. It seems reasonable to propose therefore that assessment of the difficulty of exam questions could be used to gauge the level of skills and knowledge expected at the end of a course. This paper reports the results of a study investigating the difficulty of exam questions using a subjective assessment of difficulty and a purpose-built exam question complexity classification scheme. The scheme, devised for exams in introductory programming courses, assesses the complexity of each question using six measures: external domain references, explicitness, linguistic complexity, conceptual complexity, length of code involved in the question and/or answer, and intellectual complexity (Bloom level). We apply the scheme to 20 introductory programming exam papers from five countries, and find substantial variation across the exams for all measures. Most exams include a mix of questions of low, medium, and high difficulty, although seven of the 20 have no questions of high difficulty. All of the complexity measures correlate with assessment of difficulty, indicating that the difficulty of an exam question relates to each of these more specific measures. We discuss the implications of these findings for the development of measures to assess learning standards in programming courses.

Academics and activists in the policy process : engagement with Australian media inquiries 2011-13

After its narrow re-election in June 2010, the Australian Labor government undertook a series of public inquiries into reform of Australian media, communications and copyright laws. One important driver of policy reform was the government’s commitment to building a National Broadband Network (NBN), and the implications this had for existing broadcasting and telecommunications policy, as it would constitute a major driver of convergence of media and communications access devices and content platforms. These inquiries included: the Convergence Review of media and communications legislation; the Australian Law Reform Commission (ALRC) review of the National Classification Scheme; the Independent Media Inquiry (Finkelstein Review) into Media and Media Regulation; and the ALRC review of Copyright and the Digital Economy. One unusual feature of this review process, discussed in the paper, was the degree to which academics were involved in the process, not simply as providers of expert opinion, but as review chairs seconded from their universities. This paper considers the role played by activist groups in all of these inquiries and their relationship to the various participants in the inquiries, as well as the implications of academics being engaged in such inquiries, not simply as activist-scholars, but as those primarily responsible for delivering policy review outcomes. The latter brings to the forefront issues arising in from direct engagement with governments and state agencies themselves, which challenges traditional understandings of the academic community as “critical outsiders” towards such policy processes.

Reliability and validity of query intent assessments

In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data.

Convergent media policy : the Australian case

This chapter considers the implications of convergence for media policy from three perspectives. First, it discusses what have been the traditional concerns of media policy, and the challenges it faces, from the perspectives of public interest theories, economic capture theories, and capitalist state theories. Second, it looks at what media convergence involves, and some of the dilemmas arising from convergent media policy including: (1) determining who is a media company; (2) regulatory parity between ‘old’ and ‘new’ media; (3) treatment of similar media content across different platforms; (4) distinguishing ‘big media’ from user-created content, and: (5) maintaining a distinction between media regulation and censorship of personal communication. Finally, it discusses attempts to reform media policy in light of these changes, including Australian media policy reports from 2011-12 including the Convergence Review, the Finkelstein Review of News Media, and the Australian Law Reform Commission’s National Classification Scheme Review. It concludes by arguing that ‘public interest’ approaches to media policy continue to have validity, even as they grapple with the complex question of how to understand the concept of influence in a convergent media environment.

Academics in the policy process : engagement with Australian media policy inquiries 2011-2013

This paper critically evaluates the series of inquires that the Australian Labor government undertook during 2011-2013 into reform of Australian media, communications and copyright laws. One important driver of policy reform was the government’s commitment to building a National Broadband Network (NBN), and the implications this had for existing broadcasting and telecommunications policy, as it would constitute a major driver of convergence of media and communications access devices and content platforms. These inquiries included: the Convergence Review of media and communications legislation; the Australian Law Reform Commission (ALRC) review of the National Classification Scheme; and the Independent Media Inquiry (Finkelstein Review) into Media and Media Regulation. One unusual feature of this review process was the degree to which academics were involved in the process, not simply as providers of expert opinion, but as review chairs seconded from their universities. This paper considers the role played by activist groups in all of these inquiries and their relationship to the various participants in the inquiries, as well as the implications of academics being engaged in such inquiries, not simply as activist-scholars, but as those primarily responsible for delivering policy review outcomes. The paper draws upon the concept of "policy windows" in order to better understand the context in which the inquiries took place, and their relative lack of legislative impact.

Convergent Media Policy

This chapter considers the implications of convergence for media policy from three perspectives. First, it discusses what have been the traditional concerns of media policy, and the challenges it faces, from the perspectives of public interest theories, economic capture theories, and capitalist state theories. Second, it looks at what media convergence involves, and some of the dilemmas arising from convergent media policy including: (1) determining who is a media company; (2) regulatory parity between ‘old’ and ‘new’ media; (3) treatment of similar media content across different platforms; (4) distinguishing ‘big media’ from user-created content; and (5) maintaining a distinction between media regulation and censorship of personal communication. Finally, it discusses attempts to reform media policy in light of these changes, including Australian media policy reports from 2011-12 including the Convergence Review, the Finkelstein Review of News Media, and the Australian Law Reform Commission’s National Classification Scheme Review. It concludes by arguing that ‘public interest’ approaches to media policy continue to have validity, even as they grapple with the complex question of how to understand the concept of influence in a convergent media environment.

Bitboard methods for games

Bitboards allow the efficient encoding of games for computer play and the application of fast bitwiseparallel algorithms for common game-related operations. This article describes: (1) a selection of bitboard techniques including an introduction to bitboards and bitwise operations; (2) a classification scheme that distinguishes filter, query and update methods, and; (3) a sampling of bitboard algorithms for a range of games other than chess, with notes on their performance and practical application.

Individual process management: A first step towards the conceptualization of individual activities

Understanding activities of individuals is of major importance because their actions are the main foundation of economic activity. However, there is a lack of understanding with regard to how individual activities are characterised. Thus, we develop a first conceptual classification for individual activities extending the view on business processes. The classification scheme contains personal care, education, professional work, domestic work, leisure and travel as primary activities and individual organization, procurement, information gathering and self-expression as secondary activities. We extend mainly prior literature on customer management by structuring processes of individuals independently from their status as customer. This enables new theoretical insights in the way companies can design their offers from a strategic point of view. Companies can use IPM to systematically analyze individual processes independent from specific products and services which is assumed to foster the development of innovative product and service offers.