A measurement instrument for process modeling research : development, test and procedural model

Process modeling is an emergent area of Information Systems research that is characterized through an abundance of conceptual work with little empirical research. To fill this gap, this paper reports on the development and validation of an instrument to measure user acceptance of process modeling grammars. We advance an extended model for a multi-stage measurement instrument development procedure, which incorporates feedback from both expert and user panels. We identify two main contributions: First, we provide a validated measurement instrument for the study of user acceptance of process modeling grammars, which can be used to assist in further empirical studies that investigate phenomena associated with the business process modeling domain. Second, in doing so, we describe in detail a procedural model for developing measurement instruments that ensures high levels of reliability and validity, which may assist fellow scholars in executing their empirical research.

A balanced scorecard approach to enterprise systems performance measurement

A range of influences, both technical and organisational, has encouraged the wide spread adoption of Enterprise Systems (ES). Nevertheless, there is a growing consensus that Enterprise Systems have in many cases failed to provide expected benefits. The increasing role of, and dependency on ES (and IT in general), and the ‘uncertainty’ of these large investments, have created a strong need to monitor and measure ES performance. This paper reports on a research project aimed at deriving an ‘Enterprise Systems benefits measurement instrument’. The research seeks to identify how Enterprise Systems benefits can be usefully measured, with a ‘balance’ between qualitative and quantitative factors.

Developing sustainable IT-related capabilities : instrument development and test

Research has established that firms' IT-related capabilities at a point in time explain IT-related performance differences across firms. IT resources, however, are dynamic, and evolve at an exponential rate. This means we need to understand how to sustain firms' existing capabilities to leverage opportunities offered by new IT resources. Wet suggests a higher-level resource that can sustain firms' existing IT-related capabilities. Second, we report on the development of a valid and reliable measurement instrument for measuring this higher-level resource in four stages, which includes expert feedback and a field test. The validated instrument would be useful in extending the IT business value studies to investigate how firms can sustain their IT-related capabilities. This effort will provide a deeper understanding of how firms can secure sustainable IT-related business value from their acquired IT resources.

Realizing benefits from enterprise architecture : a measurement model

Enterprise architecture (EA) management has become an intensively discussed approach to manage enterprise transformations. Despite the popularity and potential of EA, both researchers and practitioners lament a lack of knowledge about the realization of benefits from EA. To determine the benefits from EA, we explore the various dimensions of EA benefit realization and report on the development of a validated and robust measurement instrument. In this paper, we test the reliability and construct validity of the EA benefit realization model (EABRM), which we have designed based on the DeLone & McLean IS success model and findings from exploratory interviews. A confirmatory factor analysis confirms the existence of an impact of five distinct and individually important dimensions on the benefits derived from EA: EA artefact quality, EA infrastructure quality, EA service quality, EA culture, and EA use. The analysis presented in this paper shows that the EA benefit realization model is an instrument that demonstrates strong reliability and validity.

Development and validation of an instrument to measure organizational cultures' support of business process management

The purpose of Business Process Management (BPM) is to increase the efficiency and effectiveness of organizational processes through improvement and innovation. Despite a common understanding that culture is an important element in these efforts, there is a dearth of theoretical and empirical research on culture as a facilitator of successful BPM. We develop the BPM culture construct and propose a validated instrument with which to measure organizational cultures’ support of BPM. The operationalization of the BPM culture concept provides a theoretical foundation for future research and a tool to assist organizations in developing a cultural environment that supports successful BPM.

Measuring the use of information and communication technologies (ICTs) in the classroom

In 2003, the “ICT Curriculum Integration Performance Measurement Instrument” was developed froman extensive review ofthe contemporary international and Australian research pertaining to the definition and measurement of ICT curriculum integration in classrooms (Proctor, Watson, & Finger, 2003). The 45-item instrument that resulted was based on theories and methodologies identified by the literature review. This paper describes psychometric results from a large-scale evaluation of the instrument subsequently conducted, as recommended by Proctor, Watson, and Finger (2003). The resultant 20-item, two-factor instrument, now called “Learning with ICTs: Measuring ICT Use in the Curriculum,” is both statistically and theoretically robust. This paper should be read in association with the original paper published in Computers in the Schools(Proctor, Watson, & Finger, 2003) that described in detail the theoretical framework underpinning the development of the instrument.

Measuring cognitive and dispositional characteristics of creativity in elementary students

This article describes the theoretical underpinning and development of a measurement instrument that provides teachers with a tool to observe the personal creativity characteristics of individual students. The instrument was developed by compiling a list of characteristics derived from the literature to be indicative of the personal characteristics of creative people. The list was then reduced by grouping like characteristics to 9 cognitive and dispositional traits that were considered appropriate for elementary students. The 9-item instrument was then administered in 24 classrooms to 520 Year 6 and Year 7 students. Factor analysis using maximum likelihood extraction with an oblimin rotation revealed a single factor with an eigenvalue greater than 1 and accounting for 63% of the variance. All 9 items on this factor loaded at .72 or greater. The results indicated that the Creativity Checklist has very high internal consistency and is a reliable measurement instrument (a = .93).

Construction performance and organisational culture

The purpose of this paper is to summarize the outcomes of a detailed research study carried out as part of the fulfilment of a doctoral programme which examined the relationships between, and impacts of organisational culture on construction performance within a Hong Kong context. The research used a mixed methodology approach consisting of an organisational culture survey using an adapted validated and reliable measurement instrument (the Denison Organisational Culture Survey), mini-case studies in four Hong Kong construction companies and correlated the derived culture scores against performance scores measured by the Hong Kong Housing Department Performance Assessment Scoring System (PASS). The significance of the research was to advance knowledge of the importance of organisational culture strength as a performance driver in the construction industry and the further proof of the culture performance links using a set of measures of the latter which were not financially-based. The findings of the research make a contribution to theory by further validating the work by Denison (1990) and others, not only in that a successful link between organisational culture and performance was demonstrated, but it also identifies particular cultural factors in organisations that appear to be significantly responsible for achieving successful outcomes and reveals opportunities for further research into the organisational culture of construction companies Keywords: organisational culture, construction performance, business success.

“Entrepreneurial management” as a strategic choice in firm behavior : linking it with performance

Establishing the core principals of “entrepreneurial management” within an organization describes a certain strategic choice that affects a company in six dimensions, according to Stevenson (1983). Our aim is to empirically measure entrepreneurial management (it’s existence and degree) and to link this measured strategic choice (for or against) entrepreneurial management with firm performance. Our argument here is that companies that follow core principals of entrepreneurial management should outperform other more administrative firms in certain measures of strategic performance. This paper builds on an empirical investigation published by Brown, Davidson & Wiklund (2001), who have developed and tested a reliable measurement instrument for Stevenson’s definition of “entrepreneurial management” (Stevenson 1983, Stevenson & Jarillo 1990). In the first part of our paper we aim to replicate and to some extent improve this study. In the second part we link the measured degree of “entrepreneurial management” with firm performance. To our knowledge, even so Stevenson’s definition of entrepreneurial management is commonly acknowledged and Brown et al. (2001) developed a reliable instrument to empirically capture this behavioral approach to management, the construct of entrepreneurial management never before has been linked to firm performance in an empirical study. Since most papers on corporate entrepreneurship and firm performance are based on Covin & Slevin’s (1991) or Miller’s (1983) concept of entrepreneurial orientation, we contribute to the literature on corporate entrepreneurship in a novel way, given the fact that the entrepreneurial management dimensions measured in our study can theoretically and empirically be clearly distinguished from the construct of entrepreneurial orientation as defined by Covin & Selvin (1991).

Observing the relevance of an information systems success model

This paper contributes to the rigor vs. relevance debate in the Information Systems (IS) discipline. Using the Action Research methodology, this study evaluates the relevance of a rigorously validated IS evaluation model in practice. The study captures observations of operational end-users employing a market leading Enterprise System application for procurement and order fulfillment in an organization. The analysis of the observations demonstrates the broad relevance of the measurement instrument. More importantly, the study identifies several improvements and possible confusions in applying the instrument in the practice.

Validating the is-impact model in the Malaysian public sector

This research is one of several ongoing studies conducted within the IT Professional Services (ITPS) research programme at Queensland University of Technology (QUT). In 2003, ITPS introduced the IS-Impact model, a measurement model for measuring information systems success from the viewpoint of multiple stakeholders. The model, along with its instrument, is robust, simple, yet generalisable, and yields results that are comparable across time, stakeholders, different systems and system contexts. The IS-Impact model is defined as “a measure at a point in time, of the stream of net benefits from the Information System (IS), to date and anticipated, as perceived by all key-user-groups”. The model represents four dimensions, which are ‘Individual Impact’, ‘Organizational Impact’, ‘Information Quality’ and ‘System Quality’. The two Impact dimensions measure the up-to-date impact of the evaluated system, while the remaining two Quality dimensions act as proxies for probable future impacts (Gable, Sedera & Chan, 2008). To fulfil the goal of ITPS, “to develop the most widely employed model” this research re-validates and extends the IS-Impact model in a new context. This method/context-extension research aims to test the generalisability of the model by addressing known limitations of the model. One of the limitations of the model relates to the extent of external validity of the model. In order to gain wide acceptance, a model should be consistent and work well in different contexts. The IS-Impact model, however, was only validated in the Australian context, and packaged software was chosen as the IS understudy. Thus, this study is concerned with whether the model can be applied in another different context. Aiming for a robust and standardised measurement model that can be used across different contexts, this research re-validates and extends the IS-Impact model and its instrument to public sector organisations in Malaysia. The overarching research question (managerial question) of this research is “How can public sector organisations in Malaysia measure the impact of information systems systematically and effectively?” With two main objectives, the managerial question is broken down into two specific research questions. The first research question addresses the applicability (relevance) of the dimensions and measures of the IS-Impact model in the Malaysian context. Moreover, this research question addresses the completeness of the model in the new context. Initially, this research assumes that the dimensions and measures of the IS-Impact model are sufficient for the new context. However, some IS researchers suggest that the selection of measures needs to be done purposely for different contextual settings (DeLone & McLean, 1992, Rai, Lang & Welker, 2002). Thus, the first research question is as follows, “Is the IS-Impact model complete for measuring the impact of IS in Malaysian public sector organisations?” [RQ1]. The IS-Impact model is a multidimensional model that consists of four dimensions or constructs. Each dimension is represented by formative measures or indicators. Formative measures are known as composite variables because these measures make up or form the construct, or, in this case, the dimension in the IS-Impact model. These formative measures define different aspects of the dimension, thus, a measurement model of this kind needs to be tested not just on the structural relationship between the constructs but also the validity of each measure. In a previous study, the IS-Impact model was validated using formative validation techniques, as proposed in the literature (i.e., Diamantopoulos and Winklhofer, 2001, Diamantopoulos and Siguaw, 2006, Petter, Straub and Rai, 2007). However, there is potential for improving the validation testing of the model by adding more criterion or dependent variables. This includes identifying a consequence of the IS-Impact construct for the purpose of validation. Moreover, a different approach is employed in this research, whereby the validity of the model is tested using the Partial Least Squares (PLS) method, a component-based structural equation modelling (SEM) technique. Thus, the second research question addresses the construct validation of the IS-Impact model; “Is the IS-Impact model valid as a multidimensional formative construct?” [RQ2]. This study employs two rounds of surveys, each having a different and specific aim. The first is qualitative and exploratory, aiming to investigate the applicability and sufficiency of the IS-Impact dimensions and measures in the new context. This survey was conducted in a state government in Malaysia. A total of 77 valid responses were received, yielding 278 impact statements. The results from the qualitative analysis demonstrate the applicability of most of the IS-Impact measures. The analysis also shows a significant new measure having emerged from the context. This new measure was added as one of the System Quality measures. The second survey is a quantitative survey that aims to operationalise the measures identified from the qualitative analysis and rigorously validate the model. This survey was conducted in four state governments (including the state government that was involved in the first survey). A total of 254 valid responses were used in the data analysis. Data was analysed using structural equation modelling techniques, following the guidelines for formative construct validation, to test the validity and reliability of the constructs in the model. This study is the first research that extends the complete IS-Impact model in a new context that is different in terms of nationality, language and the type of information system (IS). The main contribution of this research is to present a comprehensive, up-to-date IS-Impact model, which has been validated in the new context. The study has accomplished its purpose of testing the generalisability of the IS-Impact model and continuing the IS evaluation research by extending it in the Malaysian context. A further contribution is a validated Malaysian language IS-Impact measurement instrument. It is hoped that the validated Malaysian IS-Impact instrument will encourage related IS research in Malaysia, and that the demonstrated model validity and generalisability will encourage a cumulative tradition of research previously not possible. The study entailed several methodological improvements on prior work, including: (1) new criterion measures for the overall IS-Impact construct employed in ‘identification through measurement relations’; (2) a stronger, multi-item ‘Satisfaction’ construct, employed in ‘identification through structural relations’; (3) an alternative version of the main survey instrument in which items are randomized (rather than blocked) for comparison with the main survey data, in attention to possible common method variance (no significant differences between these two survey instruments were observed); (4) demonstrates a validation process of formative indexes of a multidimensional, second-order construct (existing examples mostly involved unidimensional constructs); (5) testing the presence of suppressor effects that influence the significance of some measures and dimensions in the model; and (6) demonstrates the effect of an imbalanced number of measures within a construct to the contribution power of each dimension in a multidimensional model.

Validity and reliability of instruments designed to measure factors influencing the overuse of antibiotics

Background: Antibiotic overuse is a global public health issue that is influenced by several factors. The degree and prevalence of antibiotic overuse is difficult to measure directly. A more practical approach, such as the use of a psycho-social measurement instrument, might allow for the observation and assessment of patterns of antibiotic use. Study objective: The aim of this paper is to review the nature, validity, and reliability of measurement scales designed to measure factors associated with antibiotic misuse/overuse. Design: This study is descriptive and includes a systematic integration of the measurement scales used in the literature to measure factors associated with antibiotic misuse/overuse. The review included 70 international scientific publications from 1992 to 2010. Main results: Studies have presented scales to measure antibiotic misuse. However, the workup of these instruments is often not mentioned, or the scales are used with only early-phase validation, such as content or face validity. Other studies have discussed the reliability of these scales. However, the full validation process has not been discussed in any of the reviewed measurement scales. Conclusion: A reliable, fully validated measurement scale must be developed to assess the factors associated with the overuse of antibiotics. Identifying these factors will help to minimize the misuse of antibiotics.

Measuring the realization of benefits from enterprise architecture management

Enterprise architecture management (EAM) has become an intensively discussed approach to manage enterprise transformations. While many organizations employ EAM, a notable insecurity about the value of EAM remains. In this paper, we propose a model to measure the realization of benefits from EAM. We identify EAM success factors and EAM benefits through a comprehensive literature review and eleven explorative expert interviews. Based on our findings, we integrate the EAM success factors and benefits with the established DeLone & McLean IS success model resulting in a model that explains the realization of EAM benefits. This model aids organizations as a benchmark and framework for identifying and assessing the setup of their EAM initiatives and whether and how EAM benefits are materialized. We see our model also as a first step to gain insights in and start a discussion on the theory of EAM benefit realization.

Measuring voluntary dietary change in response to exercise : a focus on dietary fat

Traditional treatments for weight management have focussed on prescribed dietary restriction or regular exercise, or a combination of both. However recidivism for such prescribed treatments remains high, particularly among the overweight and obese. The aim of this thesis was to investigate voluntary dietary changes in the presence of prescribed mixed-mode exercise, conducted over 16 weeks. With the implementation of a single lifestyle change (exercise) it was postulated that the onerous burden of concomitant dietary and exercise compliance would be reduced, leading to voluntary lifestyle changes in such areas as diet. In addition, the failure of exercise as a single weight loss treatment has been reported to be due to compensatory energy intakes, although much of the evidence is from acute exercise studies, necessitating investigation of compensatory intakes during a long-term exercise intervention. Following 16 weeks of moderate intensity exercise, 30 overweight and obese (BMI≥25.00 kg.m-2) men and women showed small but statistically significant decreases in mean dietary fat intakes, without compensatory increases in other macronutrient or total energy intakes. Indeed total energy intakes were significantly lower for men and women following the exercise intervention, due to the decreases in dietary fat intakes. There was a risk that acceptance of the statistical validity of the small changes to dietary fat intakes may have constituted a Type 1 error, with false rejection of the Null hypothesis. Oro-sensory perceptions to changes in fat loads were therefore investigated to determine whether the measured dietary fat changes were detectable by the human palate. The ability to detect small changes in dietary fat provides sensory feedback for self-initiated dietary changes, but lean and overweight participants were unable to distinguish changes to fat loads of similar magnitudes to that measured in the exercise intervention study. Accuracy of the dietary measurement instrument was improved with the effects of random error (day-to-day variability) minimised with the use of a statistically validated 8-day, multiple-pass, 24 hour dietary recall instrument. However systematic error (underreporting) may have masked the magnitude of dietary change, particularly the reduction in dietary fat intakes. A purported biomarker (plasma Apolipoprotein A-IV) (apoA-IV) was subsequently investigated, to monitor systematic error in self-reported dietary intakes. Changes in plasma apoA-IV concentrations were directly correlated with increased and decreased changes to dietary fat intakes, suggesting that this objective marker may be a useful tool to improve the accuracy of dietary measurement in overweight and obese populations, who are susceptible to dietary underreporting.

Assessing the overuse of antibiotics in children with URTIs in Saudi Arabia : development of the parental perception on antibiotics scale (PAPA scale)

Background: Antibiotic overuse is influenced by several factors that can only be measured using a valid and reliable psychosocial measurement instrument. This study aims to establish translation and early stage validation of an instrument recently developed by this research team to measure factors influencing the overuse of antibiotics in children with upper respiratory tract infections in Saudi Arabia. Method: The content evaluation panel was composed of area experts approached using the Delphi Technique. Experts were provided with the questionnaires iteratively, on a three-round basis until consensus on the relevance of items was reached independently. Translation was achieved by adapting Brislin’s model of translation. Results: After going through the iterative process with the experts, consensus was reached to 58 items (including demographics). Experts also pointed out some issues related to ambiguity and redundancy in some items. A final Arabic version was produced from the translation process. Conclusion: This study produced preliminary validation of the developed instrument from the experts’ contributions. Then, the instrument was translated from English to Arabic. The instrument will undergo further validation steps in the future, such as construct validity.