Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

The relationship between communication and team performance : testing moderators and identifying communication profiles in established work teams

Communication is one team process factor that has received considerable research attention in the team literature. This literature provides equivocal evidence regarding the role of communication in team performance and yet, does not provide any evidence for when communication becomes important for team performance. This research program sought to address this evidence gap by a) testing task complexity and team member diversity (race diversity, gender diversity and work value diversity) as moderators of the team communication — performance relationship; and b) testing a team communication — performance model using established teams across two different task types. The functional perspective was used as the theoretical framework for operationalizing team communication activity. The research program utilised a quasi-experimental research design with participants from a large multi-national information technology company whose Head Office was based in Sydney, Australia. Participants voluntarily completed two team building exercises (a decision making and production task), and completed two online questionnaires. In total, data were collected from 1039 individuals who constituted 203 work teams. Analysis of the data revealed a small number of significant moderation effects, not all in the expected direction. However, an interesting and unexpected finding also emerged from Study One. Large and significant correlations between communication activity ratings were found across tasks, but not within tasks. This finding suggested that teams were displaying very similar profiles of communication on each task, despite the tasks having different communication requirements. Given this finding, Study Two sought to a) determine the relative importance of task versus team effects in explaining variance in team communication measures for established teams; b) determine if established teams had reliable and discernable team communication profiles and if so, c) investigate whether team communication profiles related to task performance. Multi-level modeling and repeated measures analysis of variance (ANOVA) revealed that task type did not have an effect on team communication ratings. However, teams accounted for 24% of the total variance in communication measures. Through cluster analysis, five reliable and distinct team communication profiles were identified. Consistent with the findings of the multi-level analysis and repeated measures ANOVA, teams’ profiles were virtually identical across the decision making and production tasks. A relationship between communication profile and performance was identified for the production task, although not for the decision making task. This research responds to calls in the literature for a better understanding of when communication becomes important for team performance. The moderators tested in this research were not found to have a substantive or reliable effect on the relationship between communication and performance. However, the consistency in team communication activity suggests that established teams can be characterized by their communication profiles and further, that these communication profiles may have implications for team performance. The findings of this research provide theoretical support for the functional perspective in terms of the communication – performance relationship and further support the team development literature as an explanation for the stability in team communication profiles. This research can also assist organizations to better understand the specific types of communication activity and profiles of communication that could offer teams a performance advantage.

Rivalry within and between strategic networks : an investigation of the United States Automotive Industry

As a consequence of the increased incidence of collaborative arrangements between firms, the competitive environment characterising many industries has undergone profound change. It is suggested that rivalry is not necessarily enacted by individual firms according to the traditional mechanisms of direct confrontation in factor and product markets, but rather as collaborative orchestration between a number of participants or network members. Strategic networks are recognised as sets of firms within an industry that exhibit denser strategic linkages among themselves than other firms within the same industry. Based on this, strategic networks are determined according to evidence of strategic alliances between firms comprising the industry. As a result, a single strategic network represents a group of firms closely linked according to collaborative ties. Arguably, the collective outcome of these strategic relationships engineered between firms suggest that the collaborative benefits attributed to interorganisational relationships require closer examination in respect to their propensity to influence rivalry in intraindustry environments. Derived in large from the social sciences, network theory allows for the micro and macro examination of the opportunities and constraints inherent in the structure of relationships in strategic networks, establishing a relational approach upon which the conduct and performance of firms can be more fully understood. Research to date has yet to empirically investigate the relationship between strategic networks and rivalry. The limited research that has been completed utilising a network rationale to investigate competitive patterns in contemporary industry environments has been characterised by a failure to directly measure rivalry. Further, this prior research has typically embedded investigation in industry settings dominated by technological or regulatory imperatives, such as the microprocessor and airline industries. These industries, due to the presence of such imperatives, are arguably more inclined to support the realisation of network rivalry, through subscription to prescribed technological standards (eg., microprocessor industry) or by being bound by regulatory constraints dictating operation within particular market segments (airline industry). In order to counter these weaknesses, the proposition guiding research - Are patterns of rivalry predicted by strategic network membership? – is embedded in the United States Light Vehicles Industry, an industry not dominated by technological or regulatory imperatives. Further, rivalry is directly measured and utilised in research, thus distinguishing this investigation from prior research efforts. The timeframe of investigation is 1993 – 1999, with all research data derived from secondary sources. Strategic networks were defined within the United States Light Vehicles Industry based on evidence of horizontal strategic relationships between firms comprising the industry. The measure of rivalry used to directly ascertain the competitive patterns of industry participants was derived from the traditional Herfindahl Index, modified to account for patterns of rivalry observed at the market segment level. Statistical analyses of the strategic network and rivalry constructs found little evidence to support the contention of network rivalry; indeed, greater levels of rivalry were observed between firms comprising the same strategic network than between firms participating in opposing network structures. Based on these results, patterns of rivalry evidenced in the United States Light Vehicle Industry over the period 1993 – 1999 were not found to be predicted by strategic network membership. The findings generated by this research are in contrast to current theorising in the strategic network – rivalry realm. In this respect, these findings are surprising. The relevance of industry type, in conjunction with prevailing network methodology, provides the basis upon which these findings are contemplated. Overall, this study raises some important questions in relation to the relevancy of the network rivalry rationale, establishing a fruitful avenue for further research.

Managing hostile environments : journalists and media workers : learning to survive the world's difficult, remote and hostile environments

The aim of this research is to examine the changing nature of risks that face journalists and media workers in the world's difficult, remote and hostile environments, and consider the 'adequacy' of managing hostile environment safety courses that some media organizations require prior to foreign assignments. The study utilizes several creative works and contributions to this area of analysis, which includes a documentary film production, course contributions, an emergency reference handbook, security and incident management reviews and a template for evacuation and contingency planning. The research acknowledges that employers have a 'duty of care' to personnel working in these environments, identifies the necessity for pre-deployment training and support, and provides a solution for organizations that wish to initiate a comprehensive framework to advise, monitor, protect and respond to incidents. Finally, it explores the possible development of a unique and holistic service to facilitate proactive and responsive support, in the form of a new profession of 'Editorial Logistics Officer' or 'Editorial Safety Officer' within media organizations. This area of research is vitally important to the profession, and the intended contribution is to introduce a simple and cost-efficient framework for media organizations that desire to implement pre-deployment training and field-support – as these programs save lives. The complete proactive and responsive services may be several years from implementation. However, this study demonstrates that the facilitation of Managing Hostile Environment (MHE) courses should be the minimum professional standard. These courses have saved lives in the past and they provide journalists with the tools to "cover the story, and not become the story."

Introducing a Business Process and Software Centric Approach for Enterprise System Teaching

To meet new challenges of Enterprise Systems that essentially go beyond the initial implementation, contemporary organizations seek employees with business process experts with software skills. Despite a healthy demand from the industry for such expertise, recent studies reveal that most Information Systems (IS) graduates are ill-equipped to meet the challenges of modern organizations. This paper shares insights and experiences from a course that is designed to provide a business process centric view of a market leading Enterprise System. The course, designed for both undergraduate and graduate students, uses two common business processes in a case study that employs both sequential and explorative exercises. Student feedback gained through two longitudinal surveys across two phases of the course demonstrates promising signs of the teaching approach.