Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

Towards sustainable urban futures : evaluating urban sustainability performance of the Gold coast, Australia

Creating sustainable urban environments is one of the challenging issues that need a clear vision and implementation strategies involving changes in governmental values and decision making process for local governments. Particularly, internalisation of environmental externalities of daily urban activities (e.g. manufacturing, transportation and so on) has immense importance for which local policies are formulated to provide better living conditions for the people inhabiting urban areas. Even if environmental problems are defined succinctly by various stakeholders, complicated nature of sustainability issues demand a structured evaluation strategy and well-defined sustainability parameters for efficient and effective policy making. Following this reasoning, this study involves assessment of sustainability performance of urban settings mainly focusing on environmental problems caused by rapid urban expansion and transformation. By taking into account land-use and transportation interaction, it tries to reveal how future urban developments would alter daily urban travel behaviour of people and affect the urban and natural environments. The paper introduces a grid-based indexing method developed for this research and trailed as a GIS-based decision support tool to analyse and model selected spatial and aspatial indicators of sustainability in the Gold Coast. This process reveals parameters of site specific relationship among selected indicators that are used to evaluate index-based performance characteristics of the area. The evaluation is made through an embedded decision support module by assigning relative weights to indicators. Resolution of selected grid-based unit of analysis provides insights about service level of projected urban development proposals at a disaggregate level, such as accessibility to transportation and urban services, and pollution. The paper concludes by discussing the findings including the capacity of the decision support system to assist decision-makers in determining problematic areas and developing intervention policies for sustainable outcomes of future developments.

A GIS framework for streamlining the Nature Refuge Gazettal process

Nature Refuges encompass the second largest extent of protected area estate in Queensland. Major problems exist in the data capture, map presentation, data quality and integrity of these boundaries. The spatial accuracies/inaccuracies of the Nature Refuge administrative boundaries directly influence the ability to preserve valuable ecosystems by challenging negative environmental impacts on these properties. This research work is about supporting the Nature Refuge Programs efforts to secure Queensland’s natural and cultural values on private land by utilising GIS and its advanced functionalities. The research design organizes and enters Queensland’s Nature Refuge boundaries into a spatial environment. Survey quality data collection techniques such as the Global Positioning Systems (GPS) are investigated to capture Nature Refuge boundary information. Using the concepts of map communication GIS Cartography is utilised for the protected area plan design. New spatial datasets are generated facilitating the effectiveness of investigative data analysis. The geodatabase model developed by this study adds rich GIS behaviour providing the capability to store, query, and manipulate geographic information. It provides the ability to leverage data relationships and enforces topological integrity creating savings in customization and productivity. The final phase of the research design incorporates the advanced functions of ArcGIS. These functions facilitate building spatial system models. The geodatabase and process models developed by this research can be easily modified and the data relating to mining can be replaced by other negative environmental impacts affecting the Nature Refuges. Results of the research are presented as graphs and maps providing visual evidence supporting the usefulness of GIS as means for capturing, visualising and enhancing spatial quality and integrity of Nature Refuge boundaries.

An architecture for user configurable mobile collaborative geographic applications

Geographic information is increasingly being touted for use in research and industrial projects. While the technology is now available and affordable, there is a lack of easy to use software that takes advantage of geographic information. This is an important problem because users are often researchers or scientists who have insufficient software skills, and by providing applications that are easier to use, time and financial resources can be taken from training and be better applied to the actual research and development work. A solution for this problem must cater for the user and research needs. In particular it must allow for mobile operation for fieldwork, flexibility or customisability of data input, sharing of data with other tools and collaborative capabilities for the usual teamwork environment. This thesis has developed a new architecture and data model to achieve the solution. The result is the Mobile Collaborative Annotation framework providing an implementation of the new architecture and data model. Mobile Collaborative Mapping implements the framework as a Web 2.0 mashup rich internet application and has proven to be an effective solution through its positive application to a case study with fieldwork scientists. This thesis has contributed to research into mobile computing, collaborative computing and geospatial systems by creating a simpler entry point to mobile geospatial applications, enabling simplified collaboration and providing tangible time savings.

Sensitivity of the lane change test as a measure of in-vehicle system demand

The Lane Change Test (LCT) is one of the growing number of methods developed to quantify driving performance degradation brought about by the use of in-vehicle devices. Beyond its validity and reliability, for such a test to be of practical use, it must also be sensitive to the varied demands of individual tasks. The current study evaluated the ability of several recent LCT lateral control and event detection parameters to discriminate between visual-manual and cognitive surrogate In-Vehicle Information System tasks with different levels of demand. Twenty-seven participants (mean age 24.4 years) completed a PC version of the LCT while performing visual search and math problem solving tasks. A number of the lateral control metrics were found to be sensitive to task differences, but the event detection metrics were less able to discriminate between tasks. The mean deviation and lane excursion measures were able to distinguish between the visual and cognitive tasks, but were less sensitive to the different levels of task demand. The other LCT metrics examined were less sensitive to task differences. A major factor influencing the sensitivity of at least some of the LCT metrics could be the type of lane change instructions given to participants. The provision of clear and explicit lane change instructions and further refinement of its metrics will be essential for increasing the utility of the LCT as an evaluation tool.

A comparative study of interactive rockclimbing guidebooks and conventional hardcopy guidebooks

Climbing guidebooks have been in existence ever since people started climbing cliffs for recreation. It has only been recently that these guidebooks have started to include photographs to help identification of climbs. To date, there are very few interactive guidebooks that are available online which include the ability to filter climbs and climbing areas based upon specific characteristics. Being able to interrogate a database of climbs and climbing areas by grade, style of climbing, quality of climbing,and length of climbs would be a significant addition to the guidebooks that are currently available. Integrating a fully illustrated database of climbs with open source mapping software such as Google Maps would extend the utility of current guidebooks significantly. As portable devices become more commonplace, the ability to further combine these guidebooks with GPS technology would make the location and identification of climbs much simpler. This study compares conventional hardcopy guidebooks with several online guidebooks. In addition, several Decision Support Systems are analysed to assess the ways in which Geographic Information Systems are integrated to assist in decision making. A prototype interactive guidebook was developed after presenting a survey to a group of climbers to assess what they would find useful in an online resource. This survey found that most climbers would like to see climbs represented on a map of the climbing site in order to aid in locating them. They also suggested that being able to filter climbs by various criteria would be useful. These features were subsequently integrated into the prototype. After review by several climbers it was found that this system has many benefits over conventional hardcopy guidebooks; however, it was also noted that to be even more useful further work needed to be done to improve the functionality of the prototypes. This work would include an ability to print a selection of climbs from those ranges searched.

Measuring the performance of Customs Information Systems (CIS) in Malaysia

In the scope of this study, ‘performance measurement’ includes the collection and presentation of relevant information that reflects progress in achieving organisational strategic aims and meeting the needs of stakeholders such as merchants, importers, exporters and other clients. Evidence shows that utilising information technology (IT) in customs matters supports import and export practices and ensures that supply chain management flows seamlessly. This paper briefly reviews some practical techniques for measuring performance. Its aim is to recommend a model for measuring the performance of information systems (IS): in this case, the Customs Information System (CIS) used by the Royal Malaysian Customs Department (RMCD).The study evaluates the effectiveness of CIS implementation measures in Malaysia from an IT perspective. A model based on IS theories will be used to assess the impact of CIS. The findings of this study recommend measures for evaluating the performance of CIS and its organisational impacts in Malaysia. It is also hoped that the results of the study will assist other Customs administrations evaluate the performance of their information systems.

Measuring the impact of information systems in Malaysia

Information Systems researchers have employed a diversity of sometimes inconsistent measures of IS success, seldom explicating the rationale, thereby complicating the choice for future researchers. In response to these and other issues, Gable, Sedera and Chan introduced the IS-Impact measurement model. This model represents “the stream of net benefits from the Information System (IS), to date and anticipated, as perceived by all key-user-groups”. Although the IS-Impact model was rigorously validated in previous research, there is a need to further generalise and validate it in different context. This paper reported the findings of the IS-Impact model revalidation study at four state governments in Malaysia with 232 users of a financial system that is currently being used at eleven state governments in Malaysia. Data was analysed following the guidelines for formative measurement validation using SmartPLS. Based on the PLS results, data supported the IS-Impact dimensions and measures thus confirming the validity of the IS-Impact model in Malaysia. This indicates that the IS-Impact model is robust and can be used across different context.

Online environmental information systems

In today’s information society, electronic tools, such as computer networks for the rapid transfer of data and composite databases for information storage and management, are critical in ensuring effective environmental management. In particular environmental policies and programs for federal, state, and local governments need a large volume of up-to-date information on the quality of water, air, and soil in order to conserve and protect natural resources and to carry out meteorology. In line with this, the utilization of information and communication technologies (ICTs) is crucial to preserve and improve the quality of life. In handling tasks in the field of environmental protection a range of environmental and technical information is often required for a complex and mutual decision making in a multidisciplinary team environment. In this regard e-government provides a foundation of the transformative ICT initiative which can lead to better environmental governance, better services, and increased public participation in environmental decision- making process.

Online urban information systems

Countless factors affect the inner workings of a city, so in an attempt to gain an understanding of place and making sound decisions, planners need to utilize decision support systems (DSS) or planning support systems (PSS). PSS were originally developed as DSS in academia for experimental purposes, but like many other technologies, they became one of the most innovative technologies in parallel to rapid developments in software engineering as well as developments and advances in networks and hardware. Particularly, in the last decade, the awareness of PSS have been dramatically heightened with the increasing demand for a better, more reliable and furthermore a transparent decision-making process (Klosterman, Siebert, Hoque, Kim, & Parveen, 2003). Urban planning as an act has quite different perspective from the PSS point of view. The unique nature of planning requires that spatial dimension must be considered within the context of PSS. Additionally, the rapid changes in socio-economic structure cannot be easily monitored or controlled without an effective PSS.

Application of Geographic Modeling Techniques to Quantify Spatial Access to Health Services Before and After an Acute Cardiac Event: The Cardiac ARIA Project

Background: Access to cardiac services is essential for appropriate implementation of evidence-based therapies to improve outcomes. The Cardiac Accessibility and Remoteness Index for Australia (Cardiac ARIA) aimed to derive an objective, geographic measure reflecting access to cardiac services. Methods: An expert panel defined an evidence-based clinical pathway. Using Geographic Information Systems (GIS), a numeric/alpha index was developed at two points along the continuum of care. The acute category (numeric) measured the time from the emergency call to arrival at an appropriate medical facility via road ambulance. The aftercare category (alpha) measured access to four basic services (family doctor, pharmacy, cardiac rehabilitation, and pathology services) when a patient returned to their community. Results: The numeric index ranged from 1 (access to principle referral center with cardiac catheterization service ≤ 1 hour) to 8 (no ambulance service, > 3 hours to medical facility, air transport required). The alphabetic index ranged from A (all 4 services available within 1 hour drive-time) to E (no services available within 1 hour). 13.9 million (71%) Australians resided within Cardiac ARIA 1A locations (hospital with cardiac catheterization laboratory and all aftercare within 1 hour). Those outside Cardiac 1A were over-represented by people aged over 65 years (32%) and Indigenous people (60%). Conclusion: The Cardiac ARIA index demonstrated substantial inequity in access to cardiac services in Australia. This methodology can be used to inform cardiology health service planning and the methodology could be applied to other common disease states within other regions of the world.

Cardiac ARIA Index: Measuring Accessibility to Cardiovascular Services in Rural and Remote Australia via Applied Geographic Spatial Technology

Cardiovascular disease (CVD) continues to impose a heavy burden in terms of cost, disability and death in Australia. Evidence suggests that increasing remoteness, where cardiac services are scarce, is linked to an increased risk of dying from CVD. Fatal CVD events are reported to be between 20% and 50% higher in rural areas compared to major cities. The Cardiac ARIA project, with its extensive use of geographic Information Systems (GIS), ranks each of Australia’s 20,387 urban, rural and remote population centres by accessibility to essential services or resources for the management of a cardiac event. This unique, innovative and highly collaborative project delivers a powerful tool to highlight and combat the burden imposed by cardiovascular disease (CVD) in Australia. Cardiac ARIA is innovative. It is a model that could be applied internationally and to other acute and chronic conditions such as mental health, midwifery, cancer, respiratory, diabetes and burns services. Cardiac ARIA was designed to: 1. Determine by expert panel, what were the minimal services and resources required for the management of a cardiac event in any urban, rural or remote population locations in Australia using a single patient pathway to access care. 2. Derive a classification using GIS accessibility modelling for each of Australia’s 20,387 urban, rural and remote population locations. 3. Compare the Cardiac ARIA categories and population locations with census derived population characteristics. Key findings are as follows: • In the event of a cardiac emergency, the majority of Australians had very good access to cardiac services. Approximately 71% or 13.9 million people lived within one hour of a category one hospital. • 68% of older Australians lived within one hour of a category one hospital (Principal Referral Hospital with access to Cardiac Catheterisation). • Only 40% of indigenous people lived within one hour of the category one hospital. • 16% (74000) of indigenous people lived more than one hour from a hospital. • 3% (91,000) of people 65 years of age or older lived more than one hour from any hospital or clinic. • Approximately 96%, or 19 million, of people lived within one hour of the four key services to support cardiac rehabilitation and secondary prevention. • 75% of indigenous people lived within one hour of the four cardiac rehabilitation services to support cardiac rehabilitation and secondary prevention. Fourteen percent (64,000 persons) indigenous people had poor access to the four key services to support cardiac rehabilitation and secondary prevention. • 12% (56,000) of indigenous people were more than one hour from a hospital and only had access one the four key services (usually a medical service) to support cardiac rehabilitation and secondary prevention.

Can the Cardiac ARIA Index Improve Cardiac Care for Australia's Indigenous Population?

Background: Timely access to appropriate cardiac care is critical for optimising outcomes. Our aim was to derive an objective, comparable, geographic measure reflecting access to cardiac services for Australia's 20,387 population locations. Methods: An expert panel defined a single patient care pathway. Using geographic information systems (GIS) the numeric/alpha index was modelled in two phases. The acute phase index (numeric) ranged from 1 (access to tertiary centre with PCI ≤1 h) to 8 (no ambulance service, >3 h to medical facility, air transport required). The aftercare index was modelled into 5 alphabetic categories; A (Access to general practitioner, pharmacy, cardiac rehabilitation, pathology ≤1 h) to E (no services available within 1 h). Results: Approximately 70% or 13.9 million people lived within a CardiacARIAindex category 1A location. Disparity continues in access to category 1A cardiac services for 5.8 million (30%) of all Australians, 60% of Aboriginal and Torres Strait Islander people and 32% of people over 65 years of age. In a cardiac emergency only 40% of the Indigenous population reside within one hour of category 1 hospital. Approximately 30% (81,491 Indigenous persons) are more than one to three hours from basic cardiac services. Conclusion: Geographically, the majority of Australian's have timely access for survival of a cardiac event. The CardiacARIAindex objectively demonstrates that the healthcare system may not be providing for the needs of 60% of Indigenous people residing outside the 1A geographic radius. Innovative clinical practice maybe required to address these disparities.

Information technology resources,complementaries and capabilities : towards a deeper understanding of leveraging business value from IT

Organisations devote substantial resources to acquire information technology (IT), and explaining the important issue of how IT can affect performance has posed a significant challenge to information system (IS) researchers. Owing to the importance of expanding our understanding on how and where IT and IT-related resources impact organisational performance, this study investigates the differential effects of IT resources and IT-related capabilities, in the presence of platform-related complementarities, on business process performance. We test these relationships empirically via a field survey of 216 firms. The findings suggest that IT resources and IT-related capabilities explain variance in performance. Of interest is the finding that IT resources and IT-related capabilities ability to explain variance in business process is further enhanced by the presence of the platform-related complementarities. Our findings are largely consistent with the resource-based and complementarity arguments of sources of IT-related business value.