Jigsaw Java: Intelligent, Automated Marking

The popularity of Computing degrees in the UK has been increasing significantly over the past number of years. In Northern Ireland, from 2007 to 2015, there has been a 40% increase in acceptances to Computer Science degrees with England seeing a 60% increase over the same period (UCAS, 2016). However, this is tainted as Computer Science degrees also continue to maintain the highest dropout rates.
In Queen’s University Belfast we currently have a Level 1 intake of over 400 students across a number of computing pathways. Our drive as staff is to empower and motivate the students to fully engage with the course content. All students take a Java programming module the aim of which is to provide an understanding of the basic principles of object-oriented design. In order to assess these skills, we have developed Jigsaw Java as an innovative assessment tool offering intelligent, semi-supervised automated marking of code.
Jigsaw Java allows students to answer programming questions using a drag-and-drop interface to place code fragments into position. Their answer is compared to the sample solution and if it matches, marks are allocated accordingly. However, if a match is not found then the corresponding code is executed using sample data to determine if its logic is acceptable. If it is, the solution is flagged to be checked by staff and if satisfactory is saved as an alternative solution. This means that appropriate marks can be allocated and should another student have submitted the same placement of code fragments this does not need to be executed or checked again. Rather the system now knows how to assess it.
Jigsaw Java is also able to consider partial marks dependent on code placement and will “learn” over time. Given the number of students, Jigsaw Java will improve the consistency and timeliness of marking.

Machine learning classification of surgical pathology reports and chunk recognition for information extraction noise reduction

Background and aims: Machine learning techniques for the text mining of cancer-related clinical documents have not been sufficiently explored. Here some techniques are presented for the pre-processing of free-text breast cancer pathology reports, with the aim of facilitating the extraction of information relevant to cancer staging.

Materials and methods: The first technique was implemented using the freely available software RapidMiner to classify the reports according to their general layout: ‘semi-structured’ and ‘unstructured’. The second technique was developed using the open source language engineering framework GATE and aimed at the prediction of chunks of the report text containing information pertaining to the cancer morphology, the tumour size, its hormone receptor status and the number of positive nodes. The classifiers were trained and tested respectively on sets of 635 and 163 manually classified or annotated reports, from the Northern Ireland Cancer Registry.

Results: The best result of 99.4% accuracy – which included only one semi-structured report predicted as unstructured – was produced by the layout classifier with the k nearest algorithm, using the binary term occurrence word vector type with stopword filter and pruning. For chunk recognition, the best results were found using the PAUM algorithm with the same parameters for all cases, except for the prediction of chunks containing cancer morphology. For semi-structured reports the performance ranged from 0.97 to 0.94 and from 0.92 to 0.83 in precision and recall, while for unstructured reports performance ranged from 0.91 to 0.64 and from 0.68 to 0.41 in precision and recall. Poor results were found when the classifier was trained on semi-structured reports but tested on unstructured.

Conclusions: These results show that it is possible and beneficial to predict the layout of reports and that the accuracy of prediction of which segments of a report may contain certain information is sensitive to the report layout and the type of information sought.

Recruiting faith and non-faith based schools, adolescents and parents to a cluster randomised sexual-health trial: experiences, challenges and lessons from the mixed-methods Jack Feasibility Trial

Background: The move toward evidence-based education has led to increasing numbers of randomised trials in schools. However, the literature on recruitment to non-clinical trials is relatively underdeveloped, when compared to that of clinical trials. Recruitment to school-based randomised trials is, however, challenging; even more so when the focus of the study is a sensitive issue such as sexual health. This article reflects on the challenges of recruiting post-primary schools, adolescent pupils and parents to a cluster randomised feasibility trial of a sexual health intervention, and the strategies employed to address them.
Methods: The Jack Trial was funded by the UK National Institute for Health Research (NIHR). It comprised a feasibility study of an interactive film-based sexual health intervention entitled If I Were Jack, recruiting over 800 adolescents from eight socio-demographically diverse post-primary schools in Northern Ireland. It aimed to determine the facilitators and barriers to recruitment and retention to a school-based sexual health trial and identify optimal multi-level strategies for an effectiveness study. As part of an embedded process evaluation, we conducted semi-structured interviews and focus groups with principals, vice-principals, teachers, pupils and parents recruited to the study as well as classroom observations and a parents’ survey.
Results: With reference to Social Learning Theory, we identified a number of individual, behavioural and environmental level factors which influenced recruitment. Commonly identified facilitators included perceptions of the relevance and potential benefit of the intervention to adolescents, the credibility of the organisation and individuals running the study, support offered by trial staff, and financial incentives. Key barriers were prior commitment to other research, lack of time and resources, and perceptions that the intervention was incompatible with pupil or parent needs or the school ethos.
Conclusions: Reflecting on the methodological challenges of recruiting to a school-based sexual health feasibility trial, this study highlights pertinent general and trial-specific facilitators and barriers to recruitment, which will prove useful for future trials with schools, adolescent pupils and parents.