Evaluating the success of large-scale, integrated information systems through the lens of IS-impact and IS-support

This study proceeds from a central interest in the importance of systematically evaluating operational large-scale integrated information systems (IS) in organisations. The study is conducted within the IS-Impact Research Track at Queensland University of Technology (QUT). The goal of the IS-Impact Track is, "to develop the most widely employed model for benchmarking information systems in organizations for the joint benefit of both research and practice" (Gable et al, 2009). The track espouses programmatic research having the principles of incrementalism, tenacity, holism and generalisability through replication and extension research strategies. Track efforts have yielded the bicameral IS-Impact measurement model; the ‘impact’ half includes Organisational-Impact and Individual-Impact dimensions; the ‘quality’ half includes System-Quality and Information-Quality dimensions. Akin to Gregor’s (2006) analytic theory, the ISImpact model is conceptualised as a formative, multidimensional index and is defined as "a measure at a point in time, of the stream of net benefits from the IS, to date and anticipated, as perceived by all key-user-groups" (Gable et al., 2008, p: 381). The study adopts the IS-Impact model (Gable, et al., 2008) as its core theory base. Prior work within the IS-Impact track has been consciously constrained to Financial IS for their homogeneity. This study adopts a context-extension strategy (Berthon et al., 2002) with the aim "to further validate and extend the IS-Impact measurement model in a new context - i.e. a different IS - Human Resources (HR)". The overarching research question is: "How can the impacts of large-scale integrated HR applications be effectively and efficiently benchmarked?" This managerial question (Cooper & Emory, 1995) decomposes into two more specific research questions – In the new HR context: (RQ1): "Is the IS-Impact model complete?" (RQ2): "Is the ISImpact model valid as a 1st-order formative, 2nd-order formative multidimensional construct?" The study adhered to the two-phase approach of Gable et al. (2008) to hypothesise and validate a measurement model. The initial ‘exploratory phase’ employed a zero base qualitative approach to re-instantiating the IS-Impact model in the HR context. The subsequent ‘confirmatory phase’ sought to validate the resultant hypothesised measurement model against newly gathered quantitative data. The unit of analysis for the study is the application, ‘ALESCO’, an integrated large-scale HR application implemented at Queensland University of Technology (QUT), a large Australian university (with approximately 40,000 students and 5000 staff). Target respondents of both study phases were ALESCO key-user-groups: strategic users, management users, operational users and technical users, who directly use ALESCO or its outputs. An open-ended, qualitative survey was employed in the exploratory phase, with the objective of exploring the completeness and applicability of the IS-Impact model’s dimensions and measures in the new context, and to conceptualise any resultant model changes to be operationalised in the confirmatory phase. Responses from 134 ALESCO users to the main survey question, "What do you consider have been the impacts of the ALESCO (HR) system in your division/department since its implementation?" were decomposed into 425 ‘impact citations.’ Citation mapping using a deductive (top-down) content analysis approach instantiated all dimensions and measures of the IS-Impact model, evidencing its content validity in the new context. Seeking to probe additional (perhaps negative) impacts; the survey included the additional open question "In your opinion, what can be done better to improve the ALESCO (HR) system?" Responses to this question decomposed into a further 107 citations which in the main did not map to IS-Impact, but rather coalesced around the concept of IS-Support. Deductively drawing from relevant literature, and working inductively from the unmapped citations, the new ‘IS-Support’ construct, including the four formative dimensions (i) training, (ii) documentation, (iii) assistance, and (iv) authorisation (each having reflective measures), was defined as: "a measure at a point in time, of the support, the [HR] information system key-user groups receive to increase their capabilities in utilising the system." Thus, a further goal of the study became validation of the IS-Support construct, suggesting the research question (RQ3): "Is IS-Support valid as a 1st-order reflective, 2nd-order formative multidimensional construct?" With the aim of validating IS-Impact within its nomological net (identification through structural relations), as in prior work, Satisfaction was hypothesised as its immediate consequence. The IS-Support construct having derived from a question intended to probe IS-Impacts, too was hypothesised as antecedent to Satisfaction, thereby suggesting the research question (RQ4): "What is the relative contribution of IS-Impact and IS-Support to Satisfaction?" With the goal of testing the above research questions, IS-Impact, IS-Support and Satisfaction were operationalised in a quantitative survey instrument. Partial least squares (PLS) structural equation modelling employing 221 valid responses largely evidenced the validity of the commencing IS-Impact model in the HR context. ISSupport too was validated as operationalised (including 11 reflective measures of its 4 formative dimensions). IS-Support alone explained 36% of Satisfaction; IS-Impact alone 70%; in combination both explaining 71% with virtually all influence of ISSupport subsumed by IS-Impact. Key study contributions to research include: (1) validation of IS-Impact in the HR context, (2) validation of a newly conceptualised IS-Support construct as important antecedent of Satisfaction, and (3) validation of the redundancy of IS-Support when gauging IS-Impact. The study also makes valuable contributions to practice, the research track and the sponsoring organisation.

An expanded critical incident approach for exploring information use and learning

Critical incidents offer a focus for exploratory research about human experiences, including information use and information literacy learning. This paper describes how critical incidents underpinned research about international students’ use of online information resources at two Australian universities. It outlines the development and application of an expanded critical incident approach (ECIA), explaining how ECIA built upon critical incident technique (CIT) and incorporated information literacy theory. It discusses points of expansion (differences) between CIT and ECIA. While CIT initially proved useful in structuring the research, the pilot study revealed methodological limitations. ECIA allowed more nuanced data analysis and the integration of reflection. The study produced a multifaceted word picture of international students’ experience of using online information resources to learn, and a set of critical findings about their information literacy learning needs. ECIA offers a fresh approach for researching information use, information experience, evidence-based practice, information literacy and informed learning.

International students using online information resources to learn : complex experience and learning needs

This paper reports the findings of a qualitative study which investigated 25 international students’ use of online information resources for study purposes at two Australian universities. Using an expanded critical incident approach, the study viewed international students through an information literacy lens, as information-using learners. The findings are presented in two complementary parts: as a word picture that describes their whole experience of using online information resources to learn; and as a tabulated set of critical findings that summarises their associated information literacy learning needs. The word picture shows international students’ resource use as a complex interplay of eight inter-related elements: students; information-learning environment; interactions (with online resources); strengths-challenges; learning-help; affective responses; reflective responses; cultural-linguistic dimensions. In using online resources, the international students experience an array of strengths and challenges, and an apparent information literacy imbalance between their more developed information skills and less developed critical information use. The critical findings about information literacy needs provide a framework for developing an inclusive informed learning approach that responds to international students’ complex information using experiences and needs. While the study is situated in Australia, the findings are of potential interest to educators, information professionals and researchers worldwide who seek to support learning in culturally diverse higher education contexts.

Information practices of teen content creators : the intersection of action and experiences a grounded theory study

What are the information practices of teen content creators? In the United States over two thirds of teens have participated in creating and sharing content in online communities that are developed for the purpose of allowing users to be producers of content. This study investigates how teens participating in digital participatory communities find and use information as well as how they experience the information. From this investigation emerged a model of their information practices while creating and sharing content such as film-making, visual art work, story telling, music, programming, and web site design in digital participatory communities. The research uses grounded theory methodology in a social constructionist framework to investigate the research problem: what are the information practices of teen content creators? Data was gathered through semi-structured interviews and observation of teen’s digital communities. Analysis occurred concurrently with data collection, and the principle of constant comparison was applied in analysis. As findings were constructed from the data, additional data was collected until a substantive theory was constructed and no new information emerged from data collection. The theory that was constructed from the data describes five information practices of teen content creators. The five information practices are learning community, negotiating aesthetic, negotiating control, negotiating capacity, and representing knowledge. In describing the five information practices there are three necessary descriptive components, the community of practice, the experiences of information and the information actions. The experiences of information include information as participation, inspiration, collaboration, process, and artifact. Information actions include activities that occur in the categories of gathering, thinking and creating. The experiences of information and information actions intersect in the information practices, which are situated within the specific community of practice, such as a digital participatory community. Finally, the information practices interact and build upon one another and this is represented in a graphic model and explanation.

Goodbye TAIS and thanks for all the information

The Therapeutic Advice and Information Service was funded by the National Prescribing Service to provide a national drug information service for health professionals working in the community. For ten years the service achieved high levels of client satisfaction, and reached its contracted target of 6000 enquiries about medicines per year, however the service ceased on 30 June 2010.

Creating and expressing : information as-it-is-experienced

Music and dance are art forms that involve a full mind-body experience, integrating the cognitive, affective and kinesthetic domains. To engage in creating music and dance is to use information to express oneself and communicate. In this chapter I explore the information experience of two distinct groups: those who compose music for an audience, and those who dance socially with a partner. For the composer, information sources can be a stimulus for creation. Sounds, feelings, moods, images, ideas and life experiences can trigger a creative idea. These ideas are shaped by existing musical styles and structures, and by the composer’s personal aesthetic. The intention of the composer is to communicate their expressive ideas to an audience. For the social dancer, information sources are those used to communicate with a partner. There is no intention to perform for an audience. A social dancer aims to express the music and style of the dance while creating a strong connection with their partner. Information sources include the music, the partner’s body, the emotions generated by the dance, the position of other couples on the floor and the feeling of the floor. Use of information in the arts is an under-researched experience. Most information studies are based on the assumption that information is documentary and codified. Subjective and affective information is rarely recognised and legitimised. Information-as-it-is-experienced through creative practice such as music and dance is holistic in acknowledging mind, body and spirit as well as traditional documentary forms of information. This chapter draws on empirical research to illustrate experiencing information as creating and expressing.

Information accountability and usability : are there any connections?

Availability of health information is rapidly increasing and the expansion and proliferation of health information is inevitable. The Electronic Healthcare Record, Electronic Medical Record and Personal Health Record are at the core of this trend and are required for appropriate and practicable exchange and sharing of health information. However, it is becoming increasingly recognized that it is essential to preserve patient privacy and information security when utilising sensitive information for clinical, management and administrative processes. Furthermore, the usability of emerging healthcare applications is also becoming a growing concern. This paper proposes a novel approach for integrating consideration of information accountability with a perspective from usability engineering that can be applied when developing healthcare information technology applications. A social networking user case in the healthcare information exchange will be presented in the context of our approach.

'Information in context' : co-designing workplace structures and systems for organizational learning

With the aim of advancing professional practice through better understanding how to create workplace contexts that cultivate individual and collective learning through situated 'information in context' experiences, this paper presents insights gained from three North American collaborative design (co-design) implementations. In the current project at the Auraria Library in Denver, Colorado, USA, participants use collaborative information practices to redesign face-to-face and technology-enabled communication, decision making, and planning systems. Design processes are described and results-to-date described, within an appreciative framework which values information sharing and enables knowledge creation through shared leadership.

Analyzing and defining information requirements of knowledge intensive construction management tasks on BIM

With the increasing popularity and adoption of building information modeling (BIM), the amount of digital information available about a building is overwhelming. Enormous challenges remain however in identifying meaningful and required information from a complex BIM model to support a particular construction management (CM) task. Detailed specifications of information required by different construction domains and expressive and easy-to-use BIM reasoning mechanisms are seen as an important means in addressing these challenges. This paper analyzes some of the characteristics and requirements of component-specific construction knowledge in relation to the current work practice and BIM-based applications. It is argued that domain ontologies and information extraction approaches, such as queries could significantly bring much needed support for knowledge sharing and integration of information between design, construction and facility management.

Information and Learning Experiences in the Cube: Vision, Implementation and Interaction

Executive Summary This project has commenced an exploration of learning and information experiences in the QUT Cube. Understanding learning in this environment has the potential to inform current implementations and future project development. In this report, we present early findings from the first phase of an investigation into what makes learning possible in the context of a giant interactive multi-media display such as the QUT Cube, which is an award-winning configuration that hosts several projects.

Health Information Literacy and the Experience of 65 to 79 Year Old Australians

Information literacy is presented here from a relational perspective, as people’s experience of using information to learn in a particular context. A detailed practical example of such a context is provided, in the health information literacy experience of 65–79 year old Australians. A phenomenographic investigation found five qualitatively distinct ways of experiencing health information literacy: Absorbing (intuitive reception), Targeting (a planned process), Journeying (a personal quest), Liberating (equipping for independence) and Collaborating (interacting in community). These five ways of experiencing indicated expanding awareness of context (degree of orientation towards their environment), source (breadth of esteemed information), beneficiary (the scope of people who gain) and agency (amount of activity), across HIL core aspects of information, learning and health. These results illustrate the potential contribution of relational information literacy to information science.

Conference Keynote: Information literacy research and practice: An experiential perspective

In this paper I will explore some experience-based perspectives on information literacy research and practice. The research based understanding of what information literacy looks like to those experiencing it, is very different from the standard interpretations of information literacy as involving largely text based information searching, interpretation, evaluation and use. It also involves particular understandings of the interrelation between information and learning experiences. In following this thread of the history of information literacy I will reflect on aspects of the past, present and future of information literacy research. In each of these areas I explore experiential, especially phenomenographic approaches to information literacy and information literacy education, to reveal the unfolding understanding of people’s experience of information literacy stemming from this orientation. In addressing the past I will look in particular at the contribution of the seven faces of information literacy and some lessons learned from attending to variation in experience. I will explore important directions and insights that this history may help us to retain; including the value of understanding peoples’ information literacy experience. In addressing the present, I will introduce more recent work that adopts the key ideas of informed learning by attending to both information and learning experiences in specific contexts. I will look at some contemporary directions and key issues, including the reinvention of the phenomenographic, or relational approach to information literacy as informed learning or using information to learn. I will also provide some examples of the contribution of experiential approaches to information literacy research and practice. The evolution and development of the phenomenographic approach to information literacy, and the growing attention to a dual focus on information and learning experiences in this approach will be highlighted. Finally, in addressing the future I will return to advocacy, the recognition and pursuit of the transforming and empowering heart of information literacy; and suggest that for information literacy research, including the experiential, a turn towards the emancipatory has much to offer.

A grand challenge : reflecting on university science curriculum design, collaborative partnerships and integration of information literacy skills

In 2012, Queensland University of Technology (QUT) committed to the massive project of revitalizing its Bachelor of Science (ST01) degree. Like most universities in Australia, QUT has begun work to align all courses by 2015 to the requirements of the updated Australian Qualifications Framework (AQF) which is regulated by the Tertiary Education Quality and Standards Agency (TEQSA). From the very start of the redesigned degree program, students approach scientific study with an exciting mix of theory and highly topical real world examples through their chosen “grand challenge.” These challenges, Fukushima and nuclear energy for example, are the lenses used to explore science and lead to 21st century learning outcomes for students. For the teaching and learning support staff, our grand challenge is to expose all science students to multidisciplinary content with a strong emphasis on embedding information literacies into the curriculum. With ST01, QUT is taking the initiative to rethink not only content but how units are delivered and even how we work together between the faculty, the library and learning and teaching support. This was the desired outcome but as we move from design to implementation, has this goal been achieved? A main component of the new degree is to ensure scaffolding of information literacy skills throughout the entirety of the three year course. However, with the strong focus on problem-based learning and group work skills, many issues arise both for students and lecturers. A move away from a traditional lecture style is necessary but impacts on academics’ workload and comfort levels. Therefore, academics in collaboration with librarians and other learning support staff must draw on each others’ expertise to work together to ensure pedagogy, assessments and targeted classroom activities are mapped within and between units. This partnership can counteract the tendency of isolated, unsupported academics to concentrate on day-to-day teaching at the expense of consistency between units and big picture objectives. Support staff may have a more holistic view of a course or degree than coordinators of individual units, making communication and truly collaborative planning even more critical. As well, due to staffing time pressures, design and delivery of new curriculum is generally done quickly with no option for the designers to stop and reflect on the experience and outcomes. It is vital we take this unique opportunity to closely examine what QUT has and hasn’t achieved to be able to recommend a better way forward. This presentation will discuss these important issues and stumbling blocks, to provide a set of best practice guidelines for QUT and other institutions. The aim is to help improve collaboration within the university, as well as to maximize students’ ability to put information literacy skills into action. As our students embark on their own grand challenges, we must challenge ourselves to honestly assess our own work.

Researching information experience : object and domain

Information experience has emerged as a new and dynamic field of information research in recent years. This chapter will discuss and explore information experience in two distinct ways: (a) as a research object, and; (b) as a research domain. Two recent studies will provide the context for this exploration. The first study investigated the information experiences of people using social media (e.g., Facebook, Twitter, YouTube) during natural disasters. Data was gathered by in-depth semi-structured interviews with 25 participants, from two areas affected by natural disasters (i.e., Brisbane and Townsville). The second study investigated the qualitatively different ways in which people experienced information literacy during a natural disaster. Using phenomenography, data was collected via semi-structured interviews with 7 participants. These studies represent two related yet different investigations. Taken together the studies provide a means to critically debate and reflect upon our evolving understandings of information experience, both as a research object and as a research domain. This chapter presents our preliminary reflections and concludes that further research is needed to develop and strengthen our conceptualisation of this emerging area.

Creating and sharing : teens' information practices in digital communities

Introduction In a connected world youth are participating in digital content creating communities. This paper introduces a description of teens' information practices in digital content creating and sharing communities. Method The research design was a constructivist grounded theory methodology. Seventeen interviews with eleven teens were collected and observation of their digital communities occurred over a two-year period. Analysis The data were analysed iteratively to describe teens' interactions with information through open and then focused coding. Emergent categories were shared with participants to confirm conceptual categories. Focused coding provided connections between conceptual categories resulting in the theory, which was also shared with participants for feedback. Results The paper posits a substantive theory of teens' information practices as they create and share content. It highlights that teens engage in the information actions of accessing, evaluating, and using information. They experienced information in five ways: participation, information, collaboration, process, and artefact. The intersection of enacting information actions and experiences of information resulted in five information practices: learning community, negotiating aesthetic, negotiating control, negotiating capacity, and representing knowledge. Conclusion This study contributes to our understanding of youth information actions, experiences, and practices. Further research into these communities might indicate what information practices are foundational to digital communities.