Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Evaluating the success of large-scale, integrated information systems through the lens of IS-impact and IS-support

This study proceeds from a central interest in the importance of systematically evaluating operational large-scale integrated information systems (IS) in organisations. The study is conducted within the IS-Impact Research Track at Queensland University of Technology (QUT). The goal of the IS-Impact Track is, "to develop the most widely employed model for benchmarking information systems in organizations for the joint benefit of both research and practice" (Gable et al, 2009). The track espouses programmatic research having the principles of incrementalism, tenacity, holism and generalisability through replication and extension research strategies. Track efforts have yielded the bicameral IS-Impact measurement model; the ‘impact’ half includes Organisational-Impact and Individual-Impact dimensions; the ‘quality’ half includes System-Quality and Information-Quality dimensions. Akin to Gregor’s (2006) analytic theory, the ISImpact model is conceptualised as a formative, multidimensional index and is defined as "a measure at a point in time, of the stream of net benefits from the IS, to date and anticipated, as perceived by all key-user-groups" (Gable et al., 2008, p: 381). The study adopts the IS-Impact model (Gable, et al., 2008) as its core theory base. Prior work within the IS-Impact track has been consciously constrained to Financial IS for their homogeneity. This study adopts a context-extension strategy (Berthon et al., 2002) with the aim "to further validate and extend the IS-Impact measurement model in a new context - i.e. a different IS - Human Resources (HR)". The overarching research question is: "How can the impacts of large-scale integrated HR applications be effectively and efficiently benchmarked?" This managerial question (Cooper & Emory, 1995) decomposes into two more specific research questions – In the new HR context: (RQ1): "Is the IS-Impact model complete?" (RQ2): "Is the ISImpact model valid as a 1st-order formative, 2nd-order formative multidimensional construct?" The study adhered to the two-phase approach of Gable et al. (2008) to hypothesise and validate a measurement model. The initial ‘exploratory phase’ employed a zero base qualitative approach to re-instantiating the IS-Impact model in the HR context. The subsequent ‘confirmatory phase’ sought to validate the resultant hypothesised measurement model against newly gathered quantitative data. The unit of analysis for the study is the application, ‘ALESCO’, an integrated large-scale HR application implemented at Queensland University of Technology (QUT), a large Australian university (with approximately 40,000 students and 5000 staff). Target respondents of both study phases were ALESCO key-user-groups: strategic users, management users, operational users and technical users, who directly use ALESCO or its outputs. An open-ended, qualitative survey was employed in the exploratory phase, with the objective of exploring the completeness and applicability of the IS-Impact model’s dimensions and measures in the new context, and to conceptualise any resultant model changes to be operationalised in the confirmatory phase. Responses from 134 ALESCO users to the main survey question, "What do you consider have been the impacts of the ALESCO (HR) system in your division/department since its implementation?" were decomposed into 425 ‘impact citations.’ Citation mapping using a deductive (top-down) content analysis approach instantiated all dimensions and measures of the IS-Impact model, evidencing its content validity in the new context. Seeking to probe additional (perhaps negative) impacts; the survey included the additional open question "In your opinion, what can be done better to improve the ALESCO (HR) system?" Responses to this question decomposed into a further 107 citations which in the main did not map to IS-Impact, but rather coalesced around the concept of IS-Support. Deductively drawing from relevant literature, and working inductively from the unmapped citations, the new ‘IS-Support’ construct, including the four formative dimensions (i) training, (ii) documentation, (iii) assistance, and (iv) authorisation (each having reflective measures), was defined as: "a measure at a point in time, of the support, the [HR] information system key-user groups receive to increase their capabilities in utilising the system." Thus, a further goal of the study became validation of the IS-Support construct, suggesting the research question (RQ3): "Is IS-Support valid as a 1st-order reflective, 2nd-order formative multidimensional construct?" With the aim of validating IS-Impact within its nomological net (identification through structural relations), as in prior work, Satisfaction was hypothesised as its immediate consequence. The IS-Support construct having derived from a question intended to probe IS-Impacts, too was hypothesised as antecedent to Satisfaction, thereby suggesting the research question (RQ4): "What is the relative contribution of IS-Impact and IS-Support to Satisfaction?" With the goal of testing the above research questions, IS-Impact, IS-Support and Satisfaction were operationalised in a quantitative survey instrument. Partial least squares (PLS) structural equation modelling employing 221 valid responses largely evidenced the validity of the commencing IS-Impact model in the HR context. ISSupport too was validated as operationalised (including 11 reflective measures of its 4 formative dimensions). IS-Support alone explained 36% of Satisfaction; IS-Impact alone 70%; in combination both explaining 71% with virtually all influence of ISSupport subsumed by IS-Impact. Key study contributions to research include: (1) validation of IS-Impact in the HR context, (2) validation of a newly conceptualised IS-Support construct as important antecedent of Satisfaction, and (3) validation of the redundancy of IS-Support when gauging IS-Impact. The study also makes valuable contributions to practice, the research track and the sponsoring organisation.

Content sharing on public screens: experiences through iterating social and spatial contexts

This paper describes the design and study of public urban screen applications aiming to facilitate urban dwellers to control content shown on public urban screens. Two types of content sharing are presented: aggregating existing social media content about particular locations for sharing, and sharing online videos with collocated people at a public urban screen. The paper describes an exploratory study, an observational study, as well as an interpretational study in regards to application usage and user experience. Sharing content on public urban screens can pique the curiosity of users towards collocated people and the application itself resulting in raised awareness of collocated people.

Convergence and Australian content : the importance of access

In the light of new and complex challenges to media policy and regulation, the Austrlaian government commissioned the Convergence Review in late 2010 to assess the continuing applicability and utility of the principles and objectives that have shaped the policy framework to this point. It proposed a range of options for policy change and identified three enduring priorities for continued media regulation: media ownership and control; content standards; and Australian content production and distribution. The purpose of this article is to highlight an area where we feel there are opportunities for further discussion and research: the question of how the accessibility and visibility of Australian and local content may be assured in the future media policy framework via a combination of regulation and incentives to encourage innovation in content distribution.

The Convergence Review and the future of Australian content regulation

This article examines the place of Australian and local content regulation in the new media policy framework proposed by the Convergence Review. It outlines the history of Australian content regulation and the existing policy framework, before going on to detail some of the debates around Australian content during the Review. The final section analyses the relevant recommendations in the Convergence Review Final Report, and highlights some issues and problems that may arise in the new framework.

Cyclic Imide and Open-Chain Amide Carboxylic Acid Derivatives from the Facile Reaction of cis-Cyclohexane-1,2-dicarboxylic Anhydride with Three Substituted Anilines

The structures of the compounds from the reaction of cis-cyclohexane-1,2-dicarboxylic anhydride with 4-chloroaniline [rac-N-(4-chlorophenyl)-2-carboxycycloclohexane-1-carboxamide] (1), 4-bromoaniline [2-(4-bromophenyl)-perhydroisoindolyl-1,3-dione] (2) and 3-hydroxy-4-carboxyaniline (5-aminosalicylic acid) [2-(3-hydroxy-4-carboxyphenyl)-perhydroisoindolyl-1,3-dione] (3) have been determined at 200 K. Crystals of the open-chain amide carboxylic acid 1 are orthorhombic, space group Pbcn, with unit cell dimensions a = 20.1753(10), b = 8.6267(4), c = 15.9940(9) Å, and Z = 8. Compounds 2 and 3 are cyclic imides, with 1 monoclinic having space group P21 and cell dimensions a = 11.5321(3), b = 6.7095(2), c = 17.2040(5) Å, β = 102.527(3)o. Compound 3 is orthorhombic with cell dimensions a = 6.4642(3), b = 12.8196(5), c = 16.4197(7) Å. Molecules of 1 form hydrogen-bonded cyclic dimers which are extended into a two-dimensional layered structure through amide-group associations: 3 forms into one-dimensional zigzag chains through carboxylic acid…imide O-atom hydrogen bonds, while compound 2 is essentially unassociated. With both cyclic imides 2 and 3, disorder is found which involves the presence of partial enantiomeric replacement of the cis-cyclohexane-1,2-substituted ring systems.

The pedagogical, linguistic and content features of popular English language learning websites in China : a framework for analysis and design

As increasing numbers of Chinese language learners choose to learn English online (CNNIC, 2012), there is a need to investigate popular websites and their language learning designs. This paper reports on the first stage of a study that analysed the pedagogical, linguistic and content features of 25 Chinese English Language Learning (ELL) websites ranked according to their value and importance to users. The website ranking was undertaken using a system known as PageRank. The aim of the study was to identify the features characterising popular sites as opposed to those of less popular sites for the purpose of producing a framework for ELL website design in the Chinese context. The study found that a pedagogical focus with developmental instructional materials accommodating diverse proficiency levels was a major contributor to website popularity. Chinese language use for translations and teaching directives and intermediate level English for learning materials were also significant features. Content topics included Anglophone/Western and non-Anglophone/Eastern contexts. Overall, popular websites were distinguished by their mediation of access to and scaffolded support for ELL.

The influence of oxygenated organic aerosols (OOA) and its volatile organic content on the oxidative potential of diesel particulate mater

Airborne particulate matter pollution is of concern for a number of reasons and has been widely recognised as an important risk factor to human health. A number of toxicological and epidemiological studies reported negative health effects on both respiratory and cardiovascular system. Despite the availability of a huge body of research, the underlying toxicological mechanisms by which particles induce adverse health effects are not yet entirely understood. The production of reactive oxygen species (ROS) has been shown to induce oxidative stress, which is proposed as a mechanism for many of the adverse health outcomes associated with exposure to particulate matter (PM). Therefore, it is crucial to introduce a technique that will allow rapid and routine screenings of the oxidative potential of PM.

Supporting research at QUT : a tale of three librarians and a Creative Industries super-faculty

In 2012 the existing eight disciplines of Creative Industries Faculty, QUT combined with the School of Design (formerly a component of the Faculty of Built Environment and Engineering) to create a super faculty that includes the following disciplines: Architecture, Creative Writing & Literary Studies, Dance, Drama, Fashion, Film & Television, Industrial Design, Interior Design, Journalism, Media & Communication, Landscape Architecture, Music & Sound and Urban Design. The university’s research training unit AIRS (Advanced Information Retrieval Skills) is a systematic introduction to research level information literacies. It is currently being redesigned to reflect today’s new data intensive research environment and facilitate the capacity for life-long learning. Upon completion participants are expected to be able to: 1. Demonstrate an understanding of the theory of advanced search and evaluative strategies to efficiently yield appropriate resources to create original research. 2. Apply appropriate data management strategies to organise and utilize your information proficiently, ethically and legally. 3. Identify strategies to ensure best practice in the use of information sources, information technologies, information access tools and investigative methods. All Creative Industries Faculty research students must complete this unit into which CI Librarians teach discipline specific material. The library employs a team of research specific experts as well as Liaison Librarians for each faculty. Together they develop and deliver a generic research training program that provides researcher training in the following areas: Managing Research Data, QUT ePrints: New features for tracking your research impact, Tracking Research Impact, Research Students and the Library: Overview of Library Research Support Services, Technologies for Research Collaboration, Open Access Publishing, Greater Impact via Creative Commons Licence, CAMBIA - Navigating the patent literature, Uploading Publications to QUT ePrints Workshop, AIRS for supervisors, Finding Existing Research Data, Keeping up to date:Discovering and managing current awareness information and Getting Published. In 2011 Creative Industries initiated a new faculty specific research training program to promote capacity building for research within their Faculty, with workshops designed and developed with Faculty Research Leaders, The Office of Research and Liaison Librarians. “Show me the money” which assists staff to pursue alternative funding sources was one such session that was well attended and generated much discussion and interest. Drop in support sessions for ePrints, EndNote referencing software and Tracking Research Impact for the Creative Industries were also popular options on the menu. Liaison Librarians continue to provide one-on-one consultations with individual researchers as requested. This service assists Librarians greatly with getting to know and monitoring their researchers’ changing needs. The CI Faculty has enlisted two Research Leaders, one for each of the two Schools (Design and Media, Entertainment & Creative Arts) whose role it is to mentor newer research staff. Similarly within the CI library liaison team one librarian is assigned the role of Research Coordinator, whose responsibility it is to be the primary liaison with the Assistant Dean, Research and other key Faculty research managers and is the one most likely to attend Faculty committees and meetings relating to research support.

Enabling access to and re-use of publicly funded research data as Open Educational Resources : a strategy for overcoming the legal barriers to data access and re-use

Open Educational Resources (OER) are teaching, learning and research materials that have been released under an open licence that permits online access and re-use by others. The 2012 Paris OER Declaration encourages the open licensing of educational materials produced with public funds. Digital data and data sets produced as a result of scientific and non-scientific research are an increasingly important category of educational materials. This paper discusses the legal challenges presented when publicly funded research data is made available as OER, arising from intellectual property rights, confidentiality and information privacy laws, and the lack of a legal duty to ensure data quality. If these legal challenges are not understood, addressed and effectively managed, they may impede and restrict access to and re-use of research data. This paper identifies some of the legal challenges that need to be addressed and describes 10 proposed best practices which are recommended for adoption to so that publicly funded research data can be made available for access and re-use as OER.