Evaluating information systems

The book within which this chapter appears is published as a research reference book (not a coursework textbook) on Management Information Systems (MIS) for seniors or graduate students in Chinese universities. It is hoped that this chapter, along with the others, will be helpful to MIS scholars and PhD/Masters research students in China who seek understanding of several central Information Systems (IS) research topics and related issues. The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot be addressed in its entirety in any depth within a single book chapter. The chapter proceeds from the truism that organizations have limited resources and those resources need to be invested in a way that provides greatest benefit to the organization. IT expenditure represents a substantial portion of any organization’s investment budget and IT related innovations have broad organizational impacts. Evaluation of the impact of this major investment is essential to justify this expenditure both pre- and post-investment. Evaluation is also important to prioritize possible improvements. The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational performance or the economy) or perceptions of its quality from a user perspective. This reflects the MIS emphasis – a ‘management’ emphasis rather than a software engineering emphasis2, where a software engineering emphasis might be on the technical characteristics and technical performance. Though a black-box approach limits diagnostic specificity of findings from a technical perspective, it offers many benefits. In addition to superior management information, these benefits may include economy of measurement and comparability of findings (e.g. see Part 4 on Benchmarking IS). The chapter does not purport to be a comprehensive treatment of the relevant literature. It does, however, reflect many of the more influential works, and a representative range of important writings in the area. The author has been somewhat opportunistic in Part 2, employing a single journal – The Journal of Strategic Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value from this exercise real. The chapter drills down from the general to the specific. It commences with a highlevel overview of the general topic area. This is achieved in 2 parts: - Part 1 addressing existing research in the more comprehensive IS research outlets (e.g. MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part 3, the chapter narrows to focus on the sub-topic ‘Information Systems Success Measurement’; then drilling deeper to become even more focused in Part 4 on ‘Benchmarking Information Systems’. In other words, the chapter drills down from Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4 Benchmarking IS. While the commencing Parts (1&2) are by definition broadly relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly reflect the author’s more specific interests. Thus, the three chapter foci – value of IS, measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather, each subsequent focus is in most respects a sub-set of the former. Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business Value of IS’, or the relationship between information technology and organizational performance. Part 3, ‘Information System Success Measurement’, focuses more specifically on measures and constructs employed in empirical research into the drivers of IS success (ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior measures of ISS into 6 constructs – System Quality, Information Quality, Individual Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct – Service Quality (DeLone and McLean 2003)). These 6 constructs have been used extensively, individually or in some combination, as the dependent variable in research seeking to better understand the important antecedents or drivers of IS Success. Part 3 reviews this body of work. Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more specifically on a measure of the IS that can be used as a ‘benchmark’3. This section consolidates and extends the work of the author and his colleagues4 to derive a robust, validated IS-Impact measurement model for benchmarking contemporary Information Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal research, its design and validation has emphasized its role and value as a comparator; a measure that is simple, robust and generalizable and which yields results that are as far as possible comparable across time, across stakeholders, and across differing systems and systems contexts.

Empowerment for lifelong learning : embedding information literacy into the business curriculum

Success in modern business demands effective information literacy to address the ever-changing business context. This context includes changes in Government policy reflected through legislation and regulations, developments in case law and expectations of professional associations and the public. Students require the skills to continue their own learning beyond the completion of their degree, since learning the subject content of a course alone sufficient. This paper considers the methods utilised to embed information literacy, in the context of generic skills and graduate attributes, into a Business degree’s curriculum. The paper describes how information literacy has been embedded in two sequential third-year Taxation Law courses, allowing for the explicit development of information literacy. Through the development of legal reasoning and research skills, students are empowered to continue their lifelong learning, which successful professional practice demands. The study will draw upon the experience of the course convener in designing, teaching and evaluating the courses, and on students’ experiences as illustrated through evaluation questionnaire responses and interviews. The findings of this study could be relevant to other business courses, especially company law and auditing.

Personalised ontology learning and mining for web information gathering

Over the last decade, the rapid growth and adoption of the World Wide Web has further exacerbated user needs for e±cient mechanisms for information and knowledge location, selection, and retrieval. How to gather useful and meaningful information from the Web becomes challenging to users. The capture of user information needs is key to delivering users' desired information, and user pro¯les can help to capture information needs. However, e®ectively acquiring user pro¯les is di±cult. It is argued that if user background knowledge can be speci¯ed by ontolo- gies, more accurate user pro¯les can be acquired and thus information needs can be captured e®ectively. Web users implicitly possess concept models that are obtained from their experience and education, and use the concept models in information gathering. Prior to this work, much research has attempted to use ontologies to specify user background knowledge and user concept models. However, these works have a drawback in that they cannot move beyond the subsumption of super - and sub-class structure to emphasising the speci¯c se- mantic relations in a single computational model. This has also been a challenge for years in the knowledge engineering community. Thus, using ontologies to represent user concept models and to acquire user pro¯les remains an unsolved problem in personalised Web information gathering and knowledge engineering. In this thesis, an ontology learning and mining model is proposed to acquire user pro¯les for personalised Web information gathering. The proposed compu- tational model emphasises the speci¯c is-a and part-of semantic relations in one computational model. The world knowledge and users' Local Instance Reposito- ries are used to attempt to discover and specify user background knowledge. From a world knowledge base, personalised ontologies are constructed by adopting au- tomatic or semi-automatic techniques to extract user interest concepts, focusing on user information needs. A multidimensional ontology mining method, Speci- ¯city and Exhaustivity, is also introduced in this thesis for analysing the user background knowledge discovered and speci¯ed in user personalised ontologies. The ontology learning and mining model is evaluated by comparing with human- based and state-of-the-art computational models in experiments, using a large, standard data set. The experimental results are promising for evaluation. The proposed ontology learning and mining model in this thesis helps to develop a better understanding of user pro¯le acquisition, thus providing better design of personalised Web information gathering systems. The contributions are increasingly signi¯cant, given both the rapid explosion of Web information in recent years and today's accessibility to the Internet and the full text world.

Detecting and characterising malicious executable payloads

Buffer overflow vulnerabilities continue to prevail and the sophistication of attacks targeting these vulnerabilities is continuously increasing. As a successful attack of this type has the potential to completely compromise the integrity of the targeted host, early detection is vital. This thesis examines generic approaches for detecting executable payload attacks, without prior knowledge of the implementation of the attack, in such a way that new and previously unseen attacks are detectable. Executable payloads are analysed in detail for attacks targeting the Linux and Windows operating systems executing on an Intel IA-32 architecture. The execution flow of attack payloads are analysed and a generic model of execution is examined. A novel classification scheme for executable attack payloads is presented which allows for characterisation of executable payloads and facilitates vulnerability and threat assessments, and intrusion detection capability assessments for intrusion detection systems. An intrusion detection capability assessment may be utilised to determine whether or not a deployed system is able to detect a specific attack and to identify requirements for intrusion detection functionality for the development of new detection methods. Two novel detection methods are presented capable of detecting new and previously unseen executable attack payloads. The detection methods are capable of identifying and enumerating the executable payload’s interactions with the operating system on the targeted host at the time of compromise. The detection methods are further validated using real world data including executable payload attacks.

Real time detectionof airborne fungal spores and investigations into their dynamics in indoor air

Concern regarding the health effects of indoor air quality has grown in recent years, due to the increased prevalence of many diseases, as well as the fact that many people now spend most of their time indoors. While numerous studies have reported on the dynamics of aerosols indoors, the dynamics of bioaerosols in indoor environments are still poorly understood and very few studies have focused on fungal spore dynamics in indoor environments. Consequently, this work investigated the dynamics of fungal spores in indoor air, including fungal spore release and deposition, as well as investigating the mechanisms involved in the fungal spore fragmentation process. In relation to the investigation of fungal spore dynamics, it was found that the deposition rates of the bioaerosols (fungal propagules) were in the same range as the deposition rates of nonbiological particles and that they were a function of their aerodynamic diameters. It was also found that fungal particle deposition rates increased with increasing ventilation rates. These results (which are reported for the first time) are important for developing an understanding of the dynamics of fungal spores in the air. In relation to the process of fungal spore fragmentation, important information was generated concerning the airborne dynamics of the spores, as well as the part/s of the fungi which undergo fragmentation. The results obtained from these investigations into the dynamics of fungal propagules in indoor air significantly advance knowledge about the fate of fungal propagules in indoor air, as well as their deposition in the respiratory tract. The need to develop an advanced, real-time method for monitoring bioaerosols has become increasingly important in recent years, particularly as a result of the increased threat from biological weapons and bioterrorism. However, to date, the Ultraviolet Aerodynamic Particle Sizer (UVAPS, Model 3312, TSI, St Paul, MN) is the only commercially available instrument capable of monitoring and measuring viable airborne micro-organisms in real-time. Therefore (for the first time), this work also investigated the ability of the UVAPS to measure and characterise fungal spores in indoor air. The UVAPS was found to be sufficiently sensitive for detecting and measuring fungal propagules. Based on fungal spore size distributions, together with fluorescent percentages and intensities, it was also found to be capable of discriminating between two fungal spore species, under controlled laboratory conditions. In the field, however, it would not be possible to use the UVAPS to differentiate between different fungal spore species because the different micro-organisms present in the air may not only vary in age, but may have also been subjected to different environmental conditions. In addition, while the real-time UVAPS was found to be a good tool for the investigation of fungal particles under controlled conditions, it was not found to be selective for bioaerosols only (as per design specifications). In conclusion, the UVAPS is not recommended for use in the direct measurement of airborne viable bioaerosols in the field, including fungal particles, and further investigations into the nature of the micro-organisms, the UVAPS itself and/or its use in conjunction with other conventional biosamplers, are necessary in order to obtain more realistic results. Overall, the results obtained from this work on airborne fungal particle dynamics will contribute towards improving the detection capabilities of the UVAPS, so that it is capable of selectively monitoring and measuring bioaerosols, for which it was originally designed. This work will assist in finding and/or improving other technologies capable of the real-time monitoring of bioaerosols. The knowledge obtained from this work will also be of benefit in various other bioaerosol applications, such as understanding the transport of bioaerosols indoors.

Improvement to Chinese information retrieval by incorporating word segmentation and query expansion

The increasing diversity of the Internet has created a vast number of multilingual resources on the Web. A huge number of these documents are written in various languages other than English. Consequently, the demand for searching in non-English languages is growing exponentially. It is desirable that a search engine can search for information over collections of documents in other languages. This research investigates the techniques for developing high-quality Chinese information retrieval systems. A distinctive feature of Chinese text is that a Chinese document is a sequence of Chinese characters with no space or boundary between Chinese words. This feature makes Chinese information retrieval more difficult since a retrieved document which contains the query term as a sequence of Chinese characters may not be really relevant to the query since the query term (as a sequence Chinese characters) may not be a valid Chinese word in that documents. On the other hand, a document that is actually relevant may not be retrieved because it does not contain the query sequence but contains other relevant words. In this research, we propose two approaches to deal with the problems. In the first approach, we propose a hybrid Chinese information retrieval model by incorporating word-based techniques with the traditional character-based techniques. The aim of this approach is to investigate the influence of Chinese segmentation on the performance of Chinese information retrieval. Two ranking methods are proposed to rank retrieved documents based on the relevancy to the query calculated by combining character-based ranking and word-based ranking. Our experimental results show that Chinese segmentation can improve the performance of Chinese information retrieval, but the improvement is not significant if it incorporates only Chinese segmentation with the traditional character-based approach. In the second approach, we propose a novel query expansion method which applies text mining techniques in order to find the most relevant words to extend the query. Unlike most existing query expansion methods, which generally select the highly frequent indexing terms from the retrieved documents to expand the query. In our approach, we utilize text mining techniques to find patterns from the retrieved documents that highly correlate with the query term and then use the relevant words in the patterns to expand the original query. This research project develops and implements a Chinese information retrieval system for evaluating the proposed approaches. There are two stages in the experiments. The first stage is to investigate if high accuracy segmentation can make an improvement to Chinese information retrieval. In the second stage, a text mining based query expansion approach is implemented and a further experiment has been done to compare its performance with the standard Rocchio approach with the proposed text mining based query expansion method. The NTCIR5 Chinese collections are used in the experiments. The experiment results show that by incorporating the text mining based query expansion with the hybrid model, significant improvement has been achieved in both precision and recall assessments.