Evaluating the effectiveness of Z: The claims made about CICS and where we go from here

There have been few genuine success stories about industrial use of formal methods. Perhaps the best known and most celebrated is the use of Z by IBM (in collaboration with Oxford University's Programming Research Group) during the development of CICS/ESA (version 3.1). This work was rewarded with the prestigious Queen's Award for Technological Achievement in 1992 and is especially notable for two reasons: 1) because it is a commercial, rather than safety- or security-critical, system and 2) because the claims made about the effectiveness of Z are quantitative as well as qualitative. The most widely publicized claims are: less than half the normal number of customer-reported errors and a 9% savings in the total development costs of the release. This paper provides an independent assessment of the effectiveness of using Z on CICS based on the set of public domain documents. Using this evidence, we believe that the case study was important and valuable, but that the quantitative claims have not been substantiated. The intellectual arguments and rationale for formal methods are attractive, but their widespread commercial use is ultimately dependent upon more convincing quantitative demonstrations of effectiveness. Despite the pioneering efforts of IBM and PRG, there is still a need for rigorous, measurement-based case studies to assess when and how the methods are most effective. We describe how future similar case studies could be improved so that the results are more rigorous and conclusive.

Experimental study of the effectiveness of emergency signage

Signage systems play an important role in aiding occupants during both circulation and evacuation. Despite the fact that signage systems are an important component in building wayfinding systems, there is a lack of relevant data regarding how occupants detect, interpret and use the information conveyed by emergency signage. The effectiveness of signage systems is therefore difficult to assess. In this paper we address this issue through experimentation. The experiment involved measuring the impact of a signage system on a population of 68 test subjects who were instructed to individually vacate a building as quickly as possible via any means they thought appropriate. The evacuation path involved a number of decision points at which emergency signage was available to identify the appropriate path. Through analysis of video footage and data derived from questionnaires, the number of people who saw and utilised the signage information to assist their egress is determined. The results are then incorporated within the buildingEXODUS software and used in a demonstration of agent interaction with signage systems in a hypothetical evacuation scenario.

A case study of the effectiveness of the electronic survey as a research tool for measuring students' experience of e-learning

This paper uses a case study approach to consider the effectiveness of the electronic survey as a research tool to measure the learner voice about experiences of e-learning in a particular institutional case. Two large scale electronic surveys were carried out for the Student Experience of e-Learning (SEEL) project at the University of Greenwich in 2007 and 2008, funded by the UK Higher Education Academy (HEA). The paper considers this case to argue that, although the electronic web-based survey is a convenient method of quantitative and qualitative data collection, enabling higher education institutions swiftly to capture multiple views of large numbers of students regarding experiences of e-learning, for more robust analysis, electronic survey research is best combined with other methods of in-depth qualitative data collection. The advantages and disadvantages of the electronic survey as a research method to capture student experiences of e-learning are the focus of analysis in this short paper, which reports an overview of large-scale data collection (1,000+ responses) from two electronic surveys administered to students using surveymonkey as a web-based survey tool as part of the SEEL research project. Advantages of web-based electronic survey design include flexibility, ease of design, high degree of designer control, convenience, low costs, data security, ease of access and guarantee of confidentiality combined with researcher ability to identify users through email addresses. Disadvantages of electronic survey design include the self-selecting nature of web-enabled respondent participation, which tends to skew data collection towards students who respond effectively to email invitations. The relative inadequacy of electronic surveys to capture in-depth qualitative views of students is discussed with regard to prior recommendations from the JISC-funded Learners' Experiences of e-Learning (LEX) project, in consideration of the results from SEEL in-depth interviews with students. The paper considers the literature on web-based and email electronic survey design, summing up the relative advantages and disadvantages of electronic surveys as a tool for student experience of e-learning research. The paper concludes with a range of recommendations for designing future electronic surveys to capture the learner voice on e-learning, contributing to evidence-based learning technology research development in higher education.

Law in the community: the University of Greenwich/Greenwich Association of Disabled People Centre for Independent Living (GAD) research project

The GAD Advocacy Service is funded by the London Borough of Greenwich Directorate of Neighbourhood Services; its remit to support disabled people experiencing Hate Crime, Domestic Violence and Harassment. Run by disabled personnel and giving advice to all disabled people it is unique in London. Since its inception in 2004, the Advocacy Service has been stretched to its limit - there is a need to extend the remit of the Advocacy Service to give specialist legal advice on other issues. In 2003, the CEDRM-UK project was set up in the University of Greenwich Law Department as part of the Disability Rights Promotion International Legal Education and Research Project; its objectives were firstly, to facilitate the collection of data on the effectiveness of legislation in promoting the rights of disabled persons; and secondly, to pilot new methods in teaching and training in Human Rights Law – students acquire an expertise in Human Rights Law through research into the practical application of legislation relating to civil and human rights in the daily life of the community. In July 2007, GAD and CEDRM-UK embarked on a joint project to report on the work of the Advocacy Service and to create a database to support its caseload. The 2008-9 Project team will report on their work and findings relating to facilitating equality in the workplace; the inclusion of cancer, HIV and multiple sclerosis within the legal definition of disability and the implications of the statutory duty to promote disability equality for the provision of extracurricular activities for schoolchildren. [From the Author]

Experts on e-learning: insights gained from listening to the student voice!

The Student Experience of e-Learning Laboratory (SEEL) project at the University of Greenwich was designed to explore and then implement a number of approaches to investigate learners’ experiences of using technology to support their learning. In this paper members of the SEEL team present initial findings from a University-wide survey of nearly a 1000 students. A selection of 90 ‘cameos’, drawn from the survey data, offer further insights into personal perceptions of e-learning and illustrate the diversity of students experiences. The cameos provide a more coherent picture of individual student experience based on the totality of each person’s responses to the questionnaire. Finally, extracts from follow-up case studies, based on interviews with a small number of students, allow us to ‘hear’ the student voice more clearly. Issues arising from an analysis of the data include student preferences for communication and social networking tools, views on the ‘smartness’ of their tutors’ uses of technology and perceptions of the value of e-learning. A primary finding and the focus of this paper, is that students effectively arrive at their own individualised selection, configuration and use of technologies and software that meets their perceived needs. This ‘personalisation’ does not imply that such configurations are the most efficient, nor does it automatically suggest that effective learning is occurring. SEEL reminds us that learners are individuals, who approach learning both with and without technology in their own distinctive ways. Hearing, understanding and responding to the student voice is fundamental in maximising learning effectiveness. Institutions should consider actively developing the capacity of academic staff to advise students on the usefulness of particular online tools and resources in support of learning and consider the potential benefits of building on what students already use in their everyday lives. Given the widespread perception that students tend to be ‘digital natives’ and academic staff ‘digital immigrants’ (Prensky, 2001), this could represent a considerable cultural challenge.