Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

Part-time work of high school students and impact on educational outcomes

While increasing numbers of young high school students engage in part-time work, there is no consensus about its impact on educational outcomes. Indeed this field has had a dearth of research. The present paper presents a review of recent research, primarily from Australia and the US, although it is acknowledged that there are considerable contextual differences. Suggestions for school counsellors to harness the students’ experiences to assist in educational and career decision-making are presented.

The relationship between communication and team performance : testing moderators and identifying communication profiles in established work teams

Communication is one team process factor that has received considerable research attention in the team literature. This literature provides equivocal evidence regarding the role of communication in team performance and yet, does not provide any evidence for when communication becomes important for team performance. This research program sought to address this evidence gap by a) testing task complexity and team member diversity (race diversity, gender diversity and work value diversity) as moderators of the team communication — performance relationship; and b) testing a team communication — performance model using established teams across two different task types. The functional perspective was used as the theoretical framework for operationalizing team communication activity. The research program utilised a quasi-experimental research design with participants from a large multi-national information technology company whose Head Office was based in Sydney, Australia. Participants voluntarily completed two team building exercises (a decision making and production task), and completed two online questionnaires. In total, data were collected from 1039 individuals who constituted 203 work teams. Analysis of the data revealed a small number of significant moderation effects, not all in the expected direction. However, an interesting and unexpected finding also emerged from Study One. Large and significant correlations between communication activity ratings were found across tasks, but not within tasks. This finding suggested that teams were displaying very similar profiles of communication on each task, despite the tasks having different communication requirements. Given this finding, Study Two sought to a) determine the relative importance of task versus team effects in explaining variance in team communication measures for established teams; b) determine if established teams had reliable and discernable team communication profiles and if so, c) investigate whether team communication profiles related to task performance. Multi-level modeling and repeated measures analysis of variance (ANOVA) revealed that task type did not have an effect on team communication ratings. However, teams accounted for 24% of the total variance in communication measures. Through cluster analysis, five reliable and distinct team communication profiles were identified. Consistent with the findings of the multi-level analysis and repeated measures ANOVA, teams’ profiles were virtually identical across the decision making and production tasks. A relationship between communication profile and performance was identified for the production task, although not for the decision making task. This research responds to calls in the literature for a better understanding of when communication becomes important for team performance. The moderators tested in this research were not found to have a substantive or reliable effect on the relationship between communication and performance. However, the consistency in team communication activity suggests that established teams can be characterized by their communication profiles and further, that these communication profiles may have implications for team performance. The findings of this research provide theoretical support for the functional perspective in terms of the communication – performance relationship and further support the team development literature as an explanation for the stability in team communication profiles. This research can also assist organizations to better understand the specific types of communication activity and profiles of communication that could offer teams a performance advantage.

QoS-based web service composition accommodating inter-service dependencies using minimal-conflict , hill-climbing repair genetic algorithm

In the field of semantic grid, QoS-based Web service composition is an important problem. In semantic and service rich environment like semantic grid, the emergence of context constraints on Web services is very common making the composition consider not only QoS properties of Web services, but also inter service dependencies and conflicts which are formed due to the context constraints imposed on Web services. In this paper, we present a repair genetic algorithm, namely minimal-conflict hill-climbing repair genetic algorithm, to address the Web service composition optimization problem in the presence of domain constraints and inter service dependencies and conflicts. Experimental results demonstrate the scalability and effectiveness of the genetic algorithm.

Assessment of architectural work experience by employers and students

A major element in the architectural work experience program at Queensland University of Technology is the assessment reports provided by students and employers. This paper gives an analysis of assessments submitted during the period 2000 to 2007 as viewed from a practice-base perspective. By comparing the 398 student assessments with 403 employer assessments in five specific categories over an eight year period one is able to obtain a clear understanding of the performance of the program and the relevance of its various sections for its participants that is not always obvious in a yearly analysis. In the major work experience areas there is close agreement between the student and employer assessments. However, the analysis did highlight a misunderstanding of the program’s aims by some participants. Overall the students were very positive about the program and appreciated the opportunity to work on real projects and be given a degree of responsibility for these projects. For Work Integrated Learning (WIL) practitioners this study clearly demonstrates the value of obtaining assessments from students and employers in order to establish the acceptance of a WIL program.

Service staff attitudes, organisational practices and performance drivers

We provide conceptual and empirical insights elucidating how organizational practices influence service staff attitudes and behaviors and how the latter set affects organizational performance drivers. Our analyses suggest that service organizations can enhance their performance by putting in place strategies and practices that strengthen the service-oriented behaviors of their employees and reduce their intentions to leave the organization. Improved performance is accomplished through both the delivery of high quality services (enhancing organizational effectiveness) and the maintenance of frontline staff(increasing organizational efficiency). Specifically, service-oriented business strategies in the form of organizational-level service orientation and practices in the form of training directly influence the manifest service-oriented behaviors of staff. Training also indirectly affects the intention of frontline staff to leave the organization; it increases job satisfaction, which, in turn has an impact on affective commitment. Both affective and instrumental commitment were hypothesized to reduce the intentions of frontline staff to leave the organization, however only affective commitment had a significant effect.

Will they know enough? Pre-service primary teachers' knowledge base for teaching intergrated social sciences

A significant issue in primary teacher education is developing a knowledge base which prepares teachers to teach in a range of subject areas. In Australia, the problem in primary social science education is compounded by the integrated nature of the key learning area of Studies of Society and Environment (SOSE). Recent debates on teaching integrated social sciences omit discussions on the knowledge base for teaching. In this paper, a case study approach is used to investigate primary pre-service teachers’ approaches to developing a knowledge base in designing a SOSE curriculum unit. Data from five teacher-educators who taught primary SOSE curriculum indicates that novice teachers’ subject content knowledge, as revealed through their curriculum planning, lacked a disciplinary basis. However, understanding of inquiry learning, which is fundamental to social science education, was much stronger. This paper identifies a gap in the scholarship on teaching integrated social science and illustrates the need to support and develop primary teachers’ disciplinary knowledge in teacher education.

Public sector legislation in Queensland : old or new directions

It has been 150 years since the Queensland public service was established. This paper looks back over the successive civil and public service acts in Queensland from 1859 to 2009, to examine the why the acts were passed, the changing structure of the public sector and the political justifications for the changes. It will establish how much has changed and how much has stayed the same over 150 years. Discussions regarding the success of the public service acts will be approached from an accountability perspective and will work to determine how effective the legislation has been in creating an independent and efficient public sector. The paper will demonstrate that change has occurred but some of it has turned back on itself;proposals that were rejected in the past have reappeared as fresh ideas and innovations. Finally, the paper will make conclusions as to the progress or repetition of public sector legislation in Queensland.

A practical example of a siphon at work

In this paper, some classroom experiments are described for correcting the common misconception that the operation of a siphon depends on atmospheric pressure. One experiment makes use of a chain model of a siphon and another demonstrates that flow rate is dependent on the height difference between the inflow and outflow of a siphon and not atmospheric pressure. A real-life example of the use of a siphon to refill a lake in South Australia is described, demonstrating that the siphon is not only of academic interest but has practical applications.

Open standards-based system integration for asset management decision support

Over the last decade, system integration has grown in popularity as it allows organisations to streamline business processes. Traditionally, system integration has been conducted through point-to-point solutions – as a new integration scenario requirement arises, a custom solution is built between the relevant systems. Bus-based solutions are now preferred, whereby all systems communicate via an intermediary system such as an enterprise service bus, using a common data exchange model. This research investigates the use of a common data exchange model based on open standards, specifically MIMOSA OSA-EAI, for asset management system integration. A case study is conducted that involves the integration of processes between a SCADA, maintenance decision support and work management system. A diverse number of software platforms are employed in developing the final solution, all tied together through MIMOSA OSA-EAI-based XML web services. The lessons learned from the exercise are presented throughout the paper.

Realism and control : problem-based learning programs as a data source for work-related research

Problem-based learning (PBL) is a pedagogical methodology that presents the learner with a problem to be solved to stimulate and situate learning. This paper presents key characteristics of a problem-based learning environment that determines its suitability as a data source for workrelated research studies. To date, little has been written about the availability and validity of PBL environments as a data source and its suitability for work-related research. We describe problembased learning and use a research project case study to illustrate the challenges associated with industry work samples. We then describe the PBL course used in our research case study and use this example to illustrate the key attributes of problem-based learning environments and show how the chosen PBL environment met the work-related research requirements of the research case study. We propose that the more realistic the PBL work context and work group composition, the better the PBL environment as a data source for a work-related research. The work context is more realistic when relevant and complex project-based problems are tackled in industry-like work conditions over longer time frames. Work group composition is more realistic when participants with industry-level education and experience enact specialized roles in different disciplines within a professional community.