Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

Managing hostile environments : journalists and media workers : learning to survive the world's difficult, remote and hostile environments

The aim of this research is to examine the changing nature of risks that face journalists and media workers in the world's difficult, remote and hostile environments, and consider the 'adequacy' of managing hostile environment safety courses that some media organizations require prior to foreign assignments. The study utilizes several creative works and contributions to this area of analysis, which includes a documentary film production, course contributions, an emergency reference handbook, security and incident management reviews and a template for evacuation and contingency planning. The research acknowledges that employers have a 'duty of care' to personnel working in these environments, identifies the necessity for pre-deployment training and support, and provides a solution for organizations that wish to initiate a comprehensive framework to advise, monitor, protect and respond to incidents. Finally, it explores the possible development of a unique and holistic service to facilitate proactive and responsive support, in the form of a new profession of 'Editorial Logistics Officer' or 'Editorial Safety Officer' within media organizations. This area of research is vitally important to the profession, and the intended contribution is to introduce a simple and cost-efficient framework for media organizations that desire to implement pre-deployment training and field-support – as these programs save lives. The complete proactive and responsive services may be several years from implementation. However, this study demonstrates that the facilitation of Managing Hostile Environment (MHE) courses should be the minimum professional standard. These courses have saved lives in the past and they provide journalists with the tools to "cover the story, and not become the story."

The Special Broadcasting Service after 30 years : public service media and new ways of thinking about media and citizenship

This article considers the distinctive ways in which the Special Broadcasting Service (SBS) has evolved over its history since 1980, and how it has managed competing claims to being a multicultural yet broad-appeal broadcaster, and a comprehensive yet low-cost media service. It draws attention to the challenges presented by a global rethinking of the nature of citizenship and its relationship to media, for which SBS is well placed as a leader, and the challenges of online media for traditional public service media models, where SBS has arguably been a laggard, particularly when compared with the Australian Broadcasting Corporation (ABC). It notes recent work that has been undertaken by the author with others into user-created content strategies at SBS and how its online news and current affairs services have been evolving in recent years.

User-generated content and the future of public broadcasting : a case study of the special broadcasting service

This thesis presents a case study of the Special Broadcasting Service documenting the broadcasting challenges posed by user-generated content initiatives and the work-place approach to strategies for participation. Using the action research method, the project findings reveal that limitations to resources and funding determined the scope for innovation and that the practice of executive editorial control over content was considered fundamental to fulfilling the responsibilities of the public service mandate. Media workers were overwhelmingly positive about the enhanced productive capabilities of the audience and willing to facilitate moderated interactions, however the effectiveness of these initiatives differed according to the level of skills required. This thesis demonstrates how participatory initiatives can enhance aspects of the public service remit relating to cultural diversity, the servicing of niche interests, and broader social representation, and help reinvigorate the relevance of public service broadcasting in the digitalised media sphere.

QoS-based web service composition accommodating inter-service dependencies using minimal-conflict , hill-climbing repair genetic algorithm

In the field of semantic grid, QoS-based Web service composition is an important problem. In semantic and service rich environment like semantic grid, the emergence of context constraints on Web services is very common making the composition consider not only QoS properties of Web services, but also inter service dependencies and conflicts which are formed due to the context constraints imposed on Web services. In this paper, we present a repair genetic algorithm, namely minimal-conflict hill-climbing repair genetic algorithm, to address the Web service composition optimization problem in the presence of domain constraints and inter service dependencies and conflicts. Experimental results demonstrate the scalability and effectiveness of the genetic algorithm.

Will they know enough? Pre-service primary teachers' knowledge base for teaching intergrated social sciences

A significant issue in primary teacher education is developing a knowledge base which prepares teachers to teach in a range of subject areas. In Australia, the problem in primary social science education is compounded by the integrated nature of the key learning area of Studies of Society and Environment (SOSE). Recent debates on teaching integrated social sciences omit discussions on the knowledge base for teaching. In this paper, a case study approach is used to investigate primary pre-service teachers’ approaches to developing a knowledge base in designing a SOSE curriculum unit. Data from five teacher-educators who taught primary SOSE curriculum indicates that novice teachers’ subject content knowledge, as revealed through their curriculum planning, lacked a disciplinary basis. However, understanding of inquiry learning, which is fundamental to social science education, was much stronger. This paper identifies a gap in the scholarship on teaching integrated social science and illustrates the need to support and develop primary teachers’ disciplinary knowledge in teacher education.

Is the world their oyster? The global imagination of pre-service teachers

This paper reports on a qualitative interview study with eleven pre-service primary teachers in Queensland about their career plans exploring whether and how a global imagination motivates this next generation of teachers. The study is framed within sociological theory of globalisation, with regard to the growing possibilities for international mobility for work purposes, and the new life circumstances which make this imaginable. Teaching as a profession has changed and teachers are no longer as entangled with specific systems or geographical locations anymore. International recruitment campaigns are shown to pursue these pre-service teachers during their university preparation. The analysis of the interview data reveals the kind of impact these possibilities make on how pre-service teachers imagine their career, and what other considerations enhance or limit their global imagination. The findings are used to reflect on the highly localised governance of pre-service teacher preparation and the limited state-bound imaginaries to which these pre-service teachers are unnecessarily confined in their preparation.

The knowledge exchange intermediary as service provider : a discussion and an Australian case

The critical impact of innovation on national and the global economies has been discussed at length in the literature. Economic development requires the diffusion of innovations into markets. It has long been recognised that economic growth and development depends upon a constant stream of innovations. Governments have been keenly aware of the need to ensure this flow does not dry to a trickle and have introduced many and varied industry policies and interventions to assist in seeding, supporting and diffusing innovations. In Australia, as in many countries, Government support for the transfer of knowledge especially from publicly funded research has resulted in the creation of knowledge exchange intermediaries. These intermediaries are themselves service organisations, seeking innovative service offerings for their markets. The choice for most intermediaries is generally a dichotomous one, between market-pull and technology-push knowledge exchange programmes. In this article, we undertake a case analysis of one such innovative intermediary and its flagship programme. We then compare this case with other successful intermediaries in Europe. We put forward a research proposition that the design of intermediary programmes must match the service type they offer. That is, market-pull programmes require market-pull design, in close collaboration with industry, whereas technology programmes can be problem-solving innovations where demand is latent. The discussion reflects the need for an evolution in knowledge transfer policies and programmes beyond the first generation ushered in with the US Bayh-Dole Act (1980) and Stevenson-Wydler Act (1984). The data analysed is a case study comparison of market-pull and technology-push programmes, focusing on primary and secondary socio-economic benefits (using both Australian and international comparisons).

Adiabatic compression testing - part I : historical development and evaluation of fluid dynamic processes including shock-wave considerations

Adiabatic compression testing of components in gaseous oxygen is a test method that is utilized worldwide and is commonly required to qualify a component for ignition tolerance under its intended service. This testing is required by many industry standards organizations and government agencies. This paper traces the background of adiabatic compression testing in the oxygen community and discusses the thermodynamic and fluid dynamic processes that occur during rapid pressure surges. This paper is the first of several papers by the authors on the subject of adiabatic compression testing and is presented as a non-comprehensive background and introduction.

Adiabatic compression testing - part II : background and approach to estimating severity of test methodology

Adiabatic compression testing of components in gaseous oxygen is a test method that is utilized worldwide and is commonly required to qualify a component for ignition tolerance under its intended service. This testing is required by many industry standards organizations and government agencies; however, a thorough evaluation of the test parameters and test system influences on the thermal energy produced during the test has not yet been performed. This paper presents a background for adiabatic compression testing and discusses an approach to estimating potential differences in the thermal profiles produced by different test laboratories. A “Thermal Profile Test Fixture” (TPTF) is described that is capable of measuring and characterizing the thermal energy for a typical pressure shock by any test system. The test systems at Wendell Hull & Associates, Inc. (WHA) in the USA and at the BAM Federal Institute for Materials Research and Testing in Germany are compared in this manner and some of the data obtained is presented. The paper also introduces a new way of comparing the test method to idealized processes to perform system-by-system comparisons. Thus, the paper introduces an “Idealized Severity Index” (ISI) of the thermal energy to characterize a rapid pressure surge. From the TPTF data a “Test Severity Index” (TSI) can also be calculated so that the thermal energies developed by different test systems can be compared to each other and to the ISI for the equivalent isentropic process. Finally, a “Service Severity Index” (SSI) is introduced to characterizing the thermal energy of actual service conditions. This paper is the second in a series of publications planned on the subject of adiabatic compression testing.

Critical social theory and transformative learning : evidence in pre-service teachers' service-learning reflection logs

This paper reports on the opportunities for transformational learning experienced by a group of pre-service teachers who were engaged in service-learning as a pedagogical process with a focus on reflection. Critical social theory informed the design of the reflection process as it enabled a move away from knowledge transmission toward knowledge transformation. The structured reflection log was designed to illustrate the critical social theory expectations of quality learning that teach students to think critically: ideology critique and utopian critique. Butin's lenses and a reflection framework informed by the work of Bain, Ballantyne, Mills and Lester were used in the design of the service-learning reflection log. Reported data provide evidence of transformational learning and highlight how the students critique their world and imagine how they could contribute to a better world in their work as a beginning teacher.

Service driven asset management

Local governments are service driven rather than asset driven. Understanding this distinction is critical to ensuring that community needs are appropriately addressed. Translating community needs and desires into infrastructure is a complex yet little understood process. In this paper, we look at two case studies that explore the interface between service outcomes and the specification of performance requirements for the assets. The two case studies we look at are: a public health issue resulting from inadequate public amenities in a beach resort and the prioritisation of maintenance work in a world of increasing service demands and declining funding. The case studies all use the same investment logic mapping framework to establish clear drivers as to the problem that councils are responding to in delivering their services. The key to the framework is the separation of concern between service management and asset management.