Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

Quality-aware business process management

Business Process Management (BPM) has emerged as a popular management approach in both Information Technology (IT) and management practice. While there has been much research on business process modelling and the BPM life cycle, there has been little attention given to managing the quality of a business process during its life cycle. This study addresses this gap by providing a framework for organisations to manage the quality of business processes during different phases of the BPM life cycle. This study employs a multi-method research design which is based on the design science approach and the action research methodology. During the design science phase, the artifacts to model a quality-aware business process were developed. These artifacts were then evaluated through three cycles of action research which were conducted within three large Australian-based organisations. This study contributes to the body of BPM knowledge in a number of ways. Firstly, it presents a quality-aware BPM life cycle that provides a framework on how quality can be incorporated into a business process and subsequently managed during the BPM life cycle. Secondly, it provides a framework to capture and model quality requirements of a business process as a set of measurable elements that can be incorporated into the business process model. Finally, it proposes a novel root cause analysis technique for determining the causes of quality issues within business processes.

The mobile Internet and small business : an exploratory study of needs, uses and adoption with full-adopters of technology

The technological environment in which contemporary small and medium-sized enterprises (SMEs) operate can only be described as dynamic. The exponential rate of technological change, characterised by perceived increases in the benefits associated with various technologies, shortening product life cycles and changing standards, provides for the SME a complex and challenging operational context. The primary aim of this research was to concentrate on those SMEs that had already adopted technology in order to identify their needs for the new mobile data technologies (MDT), the mobile Internet. The research design utilised a mixed approach whereby both qualitative and quantitative data was collected to address the question. Overall, the needs of these SMEs for MDT can be conceptualised into three areas where the technology will assist business practices; communication, eCommerce and security.

Factors affecting entry-level Internet technology adoption by small business in Australia : evidence from three cases

The adoption of Internet technologies by the small business sector (SMEs)The adoption of Internet technologies by the small business sector is important to their on-going survival. Yet, given the opportunities and benefits that Internet technologies can provide it has been shown that Australian small businesses are relatively slow in adopting them. This paper develops a model from recent literature on the facilitators and inhibitors to the adoption of Internet technologies by small business. Cross-case analysis of findings from three case studies are presented. Findings indicate that perceived lack of business benefit, mistrust of the IT industry and lack of understanding of Internet technologies are major inhibitors to Internet adoption by small business.

Coordinated research support services at Queensland University of Technology, Australia

Queensland University of Technology (QUT) is a large multidisciplinary university located in Brisbane, Queensland, Australia. QUT is increasing its research focus and is developing its research support services. It has adopted a model of collaboration between the Library, High Performance Computing and Research Support (HPC) and more broadly with Information Technology Services (ITS). Research support services provided by the Library include the provision of information resources and discovery services, bibliographic management software, assistance with publishing (publishing strategies, identifying high impact journals, dealing with publishers and the peer review process), citation analysis and calculating authors’ H Index. Research data management services are being developed by the Library and HPC working in collaboration. The HPC group within ITS supports research computing infrastructure, research development and engagement activities, researcher consultation, high speed computation and data storage systems , 2D/ 3D (immersive) visualisation tools, parallelisation and optimization of research codes, statistics/ data modeling training and support (both qualitative and quantitative) and support for the university’s central Access Grid collaboration facility. Development and engagement activities include participation in research grants and papers, student supervision and internships and the sponsorship, incubation and adoption of new computing technologies for research. ITS also provides other services that support research including ICT training, research infrastructure (networking, data storage, federated access and authorization, virtualization) and corporate systems for research administration. Seminars and workshops are offered to increase awareness and uptake of new and existing services. A series of online surveys on eResearch practices and skills and a number of focus groups was conducted to better inform the development of research support services. Progress towards the provision of research support is described within the context organizational frameworks; resourcing; infrastructure; integration; collaboration; change management; engagement; awareness and skills; new services; and leadership. Challenges to be addressed include the need to redeploy existing operational resources toward new research support services, supporting a rapidly growing research profile across the university, the growing need for the use and support of IT in research programs, finding capacity to address the diverse research support needs across the disciplines, operationalising new research support services following their implementation in project mode, embedding new specialist staff roles, cross-skilling Liaison Librarians, and ensuring continued collaboration between stakeholders.

Facilitating unlearning during implementation of new technology

Purpose – One of the critical issues for change management, particularly in relation to the implementation of new technologies, is the existence of prior knowledge and established mental models which may hinder change efforts. Understanding unlearning and how it might assist during organizational change is a way to address this resistance. The purpose of this paper is to present research designed to identify specific factors that facilitate unlearning. Design/methodology/approach – Drawing together issues identified as potential influencers of unlearning, a survey questionnaire was developed and administered in an Australian corporation undergoing large-scale change due to the implementation of an enterprise information system. The results were analyzed to identify specific factors that impact on unlearning. Findings – Findings from this paper identify factors that hinder or help the unlearning process during times of change including understanding the need for change, the level of organizational support and training, assessment of the change, positive experience and informal support, the organization's history of change, individual's prior outlooks, and individuals' feelings and expectations. Research limitations/implications – The use of only one organization does not allow for comparisons between organizations of different sizes, cultures or industries and therefore extension of this research is recommended. Practical implications – For practitioners, this paper provides specific elements at both the level of individuals and the organization that need to be considered for optimal unlearning during times of change. Originality/value – Previous literature on unlearning has been predominantly conceptual and theoretical. These empirical findings serve to further an earlier model based on qualitative research into potential influencers of unlearning.

Life in the 'real world': a profile of Queensland University of Technology Library and Information Science graduates.

A graduate destination survey can provide a snap shot in time of a graduate’s career progression and outcome. This paper will present the results of a Queensland University of Technology study exploring the employment outcomes of students who had completed a library and information science course from the Faculty of Information Technology between 2000 and 2008. Seventy-four graduates completed an online questionnaire administered in July 2009. The study found that 90% of the graduates surveyed were working and living in Queensland, with over three quarters living and working in Brisbane. Nearly 70% were working full-time, while only 1.4% indicating that they were unemployed and looking for work. Over 80% of the graduates identified themselves as working in “librarianship”. This study is the first step in understanding the progression and destination of QUT’s library and information science graduates. It is recommended that this survey becomes an ongoing initiative so that the results can be analysed and compared over time.

Modelling and execution of asset management process via workflow automation

An Asset Management (AM) life-cycle constitutes a set of processes that align with the development, operation and maintenance of assets, in order to meet the desired requirements and objectives of the stake holders of the business. The scope of AM is often broad within an organization due to the interactions between its internal elements such as human resources, finance, technology, engineering operation, information technology and management, as well as external elements such as governance and environment. Due to the complexity of the AM processes, it has been proposed that in order to optimize asset management activities, process modelling initiatives should be adopted. Although organisations adopt AM principles and carry out AM initiatives, most do not document or model their AM processes, let alone enacting their processes (semi-) automatically using a computer-supported system. There is currently a lack of knowledge describing how to model AM processes through a methodical and suitable manner so that the processes are streamlines and optimized and are ready for deployment in a computerised way. This research aims to overcome this deficiency by developing an approach that will aid organisations in constructing AM process models quickly and systematically whilst using the most appropriate techniques, such as workflow technology. Currently, there is a wealth of information within the individual domains of AM and workflow. Both fields are gaining significant popularity in many industries thus fuelling the need for research in exploring the possible benefits of their cross-disciplinary applications. This research is thus inspired to investigate these two domains to exploit the application of workflow to modelling and execution of AM processes. Specifically, it will investigate appropriate methodologies in applying workflow techniques to AM frameworks. One of the benefits of applying workflow models to AM processes is to adapt and enable both ad-hoc and evolutionary changes over time. In addition, this can automate an AM process as well as to support the coordination and collaboration of people that are involved in carrying out the process. A workflow management system (WFMS) can be used to support the design and enactment (i.e. execution) of processes and cope with changes that occur to the process during the enactment. So far few literatures can be found in documenting a systematic approach to modelling the characteristics of AM processes. In order to obtain a workflow model for AM processes commonalities and differences between different AM processes need to be identified. This is the fundamental step in developing a conscientious workflow model for AM processes. Therefore, the first stage of this research focuses on identifying the characteristics of AM processes, especially AM decision making processes. The second stage is to review a number of contemporary workflow techniques and choose a suitable technique for application to AM decision making processes. The third stage is to develop an intermediate ameliorated AM decision process definition that improves the current process description and is ready for modelling using the workflow language selected in the previous stage. All these lead to the fourth stage where a workflow model for an AM decision making process is developed. The process model is then deployed (semi-) automatically in a state-of-the-art WFMS demonstrating the benefits of applying workflow technology to the domain of AM. Given that the information in the AM decision making process is captured at an abstract level within the scope of this work, the deployed process model can be used as an executable guideline for carrying out an AM decision process in practice. Moreover, it can be used as a vanilla system that, once being incorporated with rich information from a specific AM decision making process (e.g. in the case of a building construction or a power plant maintenance), is able to support the automation of such a process in a more elaborated way.

A cybernetic approach to land management issues

This work is a digital version of a dissertation that was first submitted in partial fulfillment of the Degree of Doctor of Philosophy at the Queensland University of Technology (QUT) in March 1994. The work was concerned with problems of self-organisation and organisation ranging from local to global levels of hierarchy. It considers organisations as living entities from local to global things that a living entity – more particularly, an individual, a body corporate or a body politic - must know and do to maintain an existence – that is to remain viable – or to be sustainable. The term ‘land management’ as used in 1994 was later subsumed into a more general concept of ‘natural resource management’ and then merged with ideas about sustainable socioeconomic and sustainable ecological development. The cybernetic approach contains many cognitive elements of human observation, language and learning that combine into production processes. The approach tends to highlight instances where systems (or organisations) can fail because they have very little chance of succeeding. Thus there are logical necessities as well as technical possibilities in designing, constructing, operating and maintaining production systems that function reliably over extended periods. Chapter numbers and titles to the original thesis are as follows: 1. Land management as a problem of coping with complexity 2. Background theory in systems theory and cybernetic principles 3. Operationalisation of cybernetic principles in Beer’s Viable System Model 4. Issues in the design of viable cadastral surveying and mapping organisation 5. An analysis of the tendency for fragmentation in surveying and mapping organisation 6. Perambulating the boundaries of Sydney – a problem of social control under poor standards of literacy 7. Cybernetic principles in the process of legislation 8. Closer settlement policy and viability in agricultural production 9. Rate of return in leasing Crown lands

An empirical test of causal relationships of factors affecting ICT adoption for building project management : an Indian SME case study

Purpose - Building project management (BPM) requires effective coordination and collaboration between multiple project team organisations which can be achieved by real time information flow between all participants. In the present scenario, this can be achieved by the use of information communication technologies (ICT). The purpose of this paper is to present part of a research project conducted to study the causal relationships between factors affecting ICT adoption for BPM by small and medium enterprises. Design/methodology/approach - This paper discusses structural equation modelling (SEM) analysis conducted to test the causal relationships between quantitative factors. Data for quantitative analysis were gathered through a questionnaire survey conducted in the Indian construction industry. Findings - SEM analysis results help in demonstrating that an increased and matured use of ICT for general administration within the organisation would lead to: an improved ICT infrastructure within the organisation; development of electronic databases; and a staff that is confident of using information technology (IT) tools. In such a scenario, staff would use advanced software and IT technologies for project management (PM) processes and that would lead to an increased adoption of ICT for PM processes. But, for general administration also, ICT adoption would be enhanced if the organisation is interacting more with geographically separated agencies and senior management perceives that significant benefits would accrue by adoption of ICT. All the factors are inter-related and their effect cannot be maximized in isolation. Originality/value - The results provide direction to building project managements for strategically adopting the effective use of ICT within their organisations and for BPM general.

Modelling complex resource requirements in Business Process Management Systems

Real-world business processes are resource-intensive. In work environments human resources usually multitask, both human and non-human resources are typically shared between tasks, and multiple resources are sometimes necessary to undertake a single task. However, current Business Process Management Systems focus on task-resource allocation in terms of individual human resources only and lack support for a full spectrum of resource classes (e.g., human or non-human, application or non-application, individual or teamwork, schedulable or unschedulable) that could contribute to tasks within a business process. In this paper we develop a conceptual data model of resources that takes into account the various resource classes and their interactions. The resulting conceptual resource model is validated using a real-life healthcare scenario.

Knowledge management competence for Enterprise System success

This study conceptualizes, operationalises and validates the concept of Knowledge Management Competence as a four-phase multidimensional formative index. Employing survey data from 310 respondents representing 27 organizations using the SAP Enterprise System Financial module, the study results demonstrate a large, significant, positive relationship between Knowledge Management Competence and Enterprise Systems Success (ES-success, as conceived by Gable Sedera and Chan (2008)); suggesting important implications for practice. Strong evidence of the validity of Knowledge Management Competence as conceived and operationalised, too suggests potential from future research evaluating its relationships with possible antecedents and consequences.

Cultivating Innovation through Social Relationships: A Qualitative Study of Outstanding Australian Innovators in Science, Technology and the Creative Industries

In this chapter, we describe and explore social relationship patterns associated with outstanding innovation. In doing so, we draw upon the findings of 16 in-depth interviews with award-winning Australian innovators from science & technology and the creative industries. The interviews covered topics relating to various influences on individual innovation capacity and career development. We found that for all of the participants, innovation was a highly social process. Although each had been recognised individually for their innovative success, none worked in isolation. The ability to generate innovative outcomes was grounded in certain types of interaction and collaboration. We outline the distinctive features of the social relationships which seem to be important to innovation, and ask which ‘social network capabilities’ might underlie the ability to create an optimal pattern of interpersonal relationships. We discuss the implications of these findings for universities, which we argue play a key role in the development of nascent innovators.

Measuring the performance of Customs Information Systems (CIS) in Malaysia

In the scope of this study, ‘performance measurement’ includes the collection and presentation of relevant information that reflects progress in achieving organisational strategic aims and meeting the needs of stakeholders such as merchants, importers, exporters and other clients. Evidence shows that utilising information technology (IT) in customs matters supports import and export practices and ensures that supply chain management flows seamlessly. This paper briefly reviews some practical techniques for measuring performance. Its aim is to recommend a model for measuring the performance of information systems (IS): in this case, the Customs Information System (CIS) used by the Royal Malaysian Customs Department (RMCD).The study evaluates the effectiveness of CIS implementation measures in Malaysia from an IT perspective. A model based on IS theories will be used to assess the impact of CIS. The findings of this study recommend measures for evaluating the performance of CIS and its organisational impacts in Malaysia. It is also hoped that the results of the study will assist other Customs administrations evaluate the performance of their information systems.